Kubernetes nodes unhealthy but pods are still running, how do I troubleshoot?

Question

I’ve been getting familiar with Kubernetes and have a 2 node pool with a 2 pod application load-balanced.

Lately, the health checks have been failing and the application is going down.

When I run kubectl get nodes both nodes have a status of ready. when I run kubectl get pods all pods are running with 0 restarts.

How does the health check work and how can I debug/troubleshoot this issue to figure out why the nodes are becoming unhealthy so that I can resolve the issue.

John Kwiatkoski · Answer

Hi there! The LB is typically checking a specific nodePort service not just checking whether a node is healthy. I understand this can be confusing with the current iteration of the LB UI. The externaltrafficpolicy setting on some kubernetes services may be set to “Local”. This means a node without a pod running locally for that service will reject the LB healthcheck and show as down. In other words, to save on overhead from unnecessary network hops between nodes, only nodes hosting pods for the LoadBalancer service will report as healthy. With externaltrafficpolicy set to “cluster”, all nodes forward that traffic to other nodes that are hosting pods for that service. In this case even a node not hosting a pod for that particular service would show as “Healthy” as it would then just forward it to a node that can service the request. To change this setting for a particular service use the following command: kubectl patch svc myservice -p ‘{“spec”:{“externalTrafficPolicy”:“Cluster”}}’ An important thing to note here is that if using the externaltrafficpolicy of “Cluster” you will lose the original client IP address due to this extra network hop. So if your application is checking for or dependent on knowing the client IP, the “Local” setting is required. You can find more information on externaltrafficpolicy here: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport Regards, John KwiatkoskiSenior Developer Support Engineer - Kubernetes

Nabeel · Answer

I had this problem a while back, and at least for me the problem was that the both instances of the ingress service I was using (nginx) were running on the same node. Because of that one node was responding and one was not. To check if you’re having the same problem, you can check how many instances of your ingress controller are running and on which nodes. If this is your problem, I solved it by adding this to my deployment spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: In values: - nginx-ingress-pod topologyKey: "kubernetes.io/hostname" However, I think the ideal solution would be to run the the service as a daemon set: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ (instead of running it as a deployment) I haven’t made that change yet myself, but would like to at some point.

Become a contributor for community

DigitalOcean Documentation

Resources for startups and AI-native businesses

The developer cloud

Start building today

Report this

Kubernetes nodes unhealthy but pods are still running, how do I troubleshoot?

Related questions

Why the Digital Ocean K8s Cluster provisioning is taking long ( 4 hours and still in progress )

Why does the Digital Ocean K8s take long to provision the cluster ( 4 hrs and still in progress )

Cilium DNS-based egress rules don’t work on DOKS

How do I add a manula droplet as node to a managed Kubernetes cluster?

How to retrieve the original IP from a HTTP request when using Cilium Gateway API on DOKS

How to get original request IP when using Cilium Gateway API on DOKS?

Kubernetes routing agent and dest: 0.0.0.0/0

Kubernetes cilium GatewayClass stuck in pending state

Bitnami migration?

How to use routing agent with nginx ingress controller ?

Problems with odoo and wkhtmltopdf in Kubernetes

Kubernetes OIDC authentication?

Report this

Kubernetes nodes unhealthy but pods are still running, how do I troubleshoot?

Related questions

Why the Digital Ocean K8s Cluster provisioning is taking long ( 4 hours and still in progress )

Why does the Digital Ocean K8s take long to provision the cluster ( 4 hrs and still in progress )

Cilium DNS-based egress rules don’t work on DOKS

How do I add a manula droplet as node to a managed Kubernetes cluster?

How to retrieve the original IP from a HTTP request when using Cilium Gateway API on DOKS

How to get original request IP when using Cilium Gateway API on DOKS?

Kubernetes routing agent and dest: 0.0.0.0/0

Kubernetes cilium GatewayClass stuck in pending state

Bitnami migration?

How to use routing agent with nginx ingress controller ?

Problems with odoo and wkhtmltopdf in Kubernetes

Kubernetes OIDC authentication?