We have the following setup:

load-balancer/Ingress -> varnish -> service -> backends

What we see is that requests from varnish to the service (service as in “Kubernetes Service”) get routed to the same backend POD over and over again (maybe 99% of the time, that is less than 1 out of 100 requests get routed to a different backend POD).

In theory the kube-proxy provided by DO should chose a backend to route the IP to randomly, since DO are using the “iptables” variant of kube-proxy. However that’s not the case at all. The request nearly always gets routed to the same backend PODs.

The result is, that that one backend POD gets overwhelmed with requests and we need to do scaling inside the POD as opposed to using the HorizontalAutoscaler which would be best practice and which will automatically launch new PODs depending on load. However those new PODs will not get routed to by the service proxy (kube-proxy as I assume).

I am out of my depths. Since I have no influence over the configuration of kube-proxy, it being provided by DO, I have no influence on how the requests going to the service IP address get distributed by kube-proxy.

What can I do? Pointers?
*t

PS: I can provide configs, but out setup being quite large, I don’t know which would be relevant here. I do not want to post all the config.