Why is Cluster Autoscaler not scaling down nodes?

I have a simple K8S cluster set up with the autoscaler enabled and one site set up with an HPA. When I run a load test on the site I can see more and more pods being provisioned and the cluster slowly scaling up nodes - in my test it scaled up to a total of 8 nodes.

The problem is that even though the HPA has brought the pods back down to from 20 to 2, it has been several hours and there is no sign of the autoscaler scaling down the cluster again. The only pods running on the new nodes are kube-system daemon sets (do-node-agent, csi-do-node, cpc-bridge-proxy, konnectivity-agent, cilium, kube-proxy)

Can someone point me in the right direction? I have been trying to find an answer but have gotten nowhere useful. I thought that daemon sets were ignored by the autoscaler in scale down decisions. Apologies in advance if this is a simple problem to resolve - I am new to K8S.