Question

Adding a node to k8s cluster causes LoadBalancer issue

Posted March 10, 2021 293 views
KubernetesDigitalOcean Managed KubernetesDigitalOcean Managed Load BalancersDigitalOcean Droplets

I just added a second node to my k8s cluster (previously I had only one to test everything out).

Now my LoadBalancers health states there is an issue and when checking the LoadBalancer it say that the second droplet is down. Checking inside k8s I can access that droplet via cli and everything else seems to be just fine.

What is the issue here? I also cannot seem to 1 click install OpenFaas. It always states that there went something wrong.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
3 answers

I just realized that the ingress pod is running on the first droplet, not the second. Does the ingress service have to run on each node?

Hi @JohannesKlauss

The LB is typically checking a specific nodePort service not just checking whether a node is healthy. To save on overhead from unnecessary network hops between nodes, only nodes hosting pods for the LoadBalancer service will report as healthy.

When the externalnetworkpolicy setting on kubernetes services is set to “Local”. This means a node without a pod running locally for that service will reject the LB healthcheck and show as down.

externaltrafficpolicy set to “cluster” would allow the nodes to forward that traffic to other nodes that are hosting pods for that service. In this case even a node not hosting a pod for that particular service would show as “UP” as it would then just forward it to a node that can, including that extra network hop.

To change this setting for particular service uses the following command:

kubectl patch svc myservice -p ’{“spec”:{“externalTrafficPolicy”:“Cluster”}}’

An important thing to note here is that if using the externaltrafficpolicy of “cluster” you will lose the original client IP address due to this extra network hop. So if your application is checking for or dependent on knowing the client IP, the “Local” setting is required.

You can find more information on externaltrafficpolicy here: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport

I hope this helps!

Best Regards,
Purnima Kumari

Hi Purnima,
thank you very much for your help. Since the server IP shouldn’t matter to my clients it’s all good. I changed it to Cluster and everything is running smooth.