Kubernetes nodes unhealthy but pods are still running, how do I troubleshoot?

Hi there!

The LB is typically checking a specific nodePort service not just checking whether a node is healthy. I understand this can be confusing with the current iteration of the LB UI.

The externaltrafficpolicy setting on some kubernetes services may be set to “Local”. This means a node without a pod running locally for that service will reject the LB healthcheck and show as down. In other words, to save on overhead from unnecessary network hops between nodes, only nodes hosting pods for the LoadBalancer service will report as healthy.

With externaltrafficpolicy set to “cluster”, all nodes forward that traffic to other nodes that are hosting pods for that service. In this case even a node not hosting a pod for that particular service would show as “Healthy” as it would then just forward it to a node that can service the request.

To change this setting for a particular service use the following command:

kubectl patch svc myservice -p ‘{“spec”:{“externalTrafficPolicy”:“Cluster”}}’

An important thing to note here is that if using the externaltrafficpolicy of “Cluster” you will lose the original client IP address due to this extra network hop. So if your application is checking for or dependent on knowing the client IP, the “Local” setting is required.

You can find more information on externaltrafficpolicy here: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport

Regards,

John Kwiatkoski Senior Developer Support Engineer - Kubernetes

I had this problem a while back, and at least for me the problem was that the both instances of the ingress service I was using (nginx) were running on the same node. Because of that one node was responding and one was not.

To check if you’re having the same problem, you can check how many instances of your ingress controller are running and on which nodes.

If this is your problem, I solved it by adding this to my deployment spec:

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - nginx-ingress-pod
              topologyKey: "kubernetes.io/hostname"

However, I think the ideal solution would be to run the the service as a daemon set: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ (instead of running it as a deployment)

I haven’t made that change yet myself, but would like to at some point.