round robin from service to backend via kube-proxy not working at DO?

Question

We have the following setup:

load-balancer/Ingress -> varnish -> service -> backends

What we see is that requests from varnish to the service (service as in “Kubernetes Service”) get routed to the same backend POD over and over again (maybe 99% of the time, that is less than 1 out of 100 requests get routed to a different backend POD).

In theory the kube-proxy provided by DO should chose a backend to route the IP to randomly, since DO are using the “iptables” variant of kube-proxy. However that’s not the case at all. The request nearly always gets routed to the same backend PODs.

The result is, that that one backend POD gets overwhelmed with requests and we need to do scaling inside the POD as opposed to using the HorizontalAutoscaler which would be best practice and which will automatically launch new PODs depending on load. However those new PODs will not get routed to by the service proxy (kube-proxy as I assume).

I am out of my depths. Since I have no influence over the configuration of kube-proxy, it being provided by DO, I have no influence on how the requests going to the service IP address get distributed by kube-proxy.

What can I do? Pointers? *t

PS: I can provide configs, but out setup being quite large, I don’t know which would be relevant here. I do not want to post all the config.

zotkinkv · Answer

We have similar problem on the recently created environment - all traffic is redirected to the single pod all the time no meter what load we have. However, service discovers pods successfully and if one of the pods fails traffic got redirected to other pod.

Has anyone solved this problem ?

Here is our setup:

Service (Also tried to create with type: Loadbalancer) :

apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  ports:
    - port: 80
      targetPort: 8080

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: default
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
    certmanager.k8s.io/cluster-issuer: "letsencrypt"
    certmanager.k8s.io/acme-challenge-type: "http01"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/websocket-services: "<Service name>"
    nginx.org/websocket-services: "notifications-service"
    nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_token"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
spec:
  tls:
  - hosts:
    - <HOST>
    secretName: letsencrypt-prod-cert
  rules:
  - host: <HOST>
    http:
      paths:
      - path: /
        backend:
          serviceName: api
          servicePort: 80

Kube-proxy configured to use iptables

maelstrome26 · Answer

I appear to also be getting this. It appears that the nodes aren’t being rotated via round robin anywhere nearly as often as they should be. Subsequent requests should be split evenly across each Node backend, but it appears they’re not being frequently.

Edit: It appears for the port you have to use the HTTP port type. Using TCP doesn’t make it work, however HTTP reliabably forwards across node backends. I edited that in the Digital Ocean console and suddenly it all started to work.

Related

Question

round robin from service to backend via kube-proxy not working at DO?

Related