Question

Deploying ingress-nginx on new cluster creates load balancer that fails health check

When I deploy ingress-nginx, a load balancer is created that points to two nodes. One healthy and one down.

Using this command: kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.40.2/deploy/static/provider/do/deploy.yaml

Following this guide: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes

Do I need to configure this further? Not sure where the problem is, or how one is successful and one fails. I’ve done this many times with the same results.

Steps to reproduce:

  1. Create a cluster (I did 2 nodes)
  2. run kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.40.2/deploy/static/provider/do/deploy.yaml
  3. Check new load balancer

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

Deploying ingress-nginx on new cluster creates load balancer that fails health check When I deploy ingress-nginx, a load balancer is created that points to two nodes. One healthy and one down.

I had the same issue when deploying a brand new kubernetes with AWS EKS using terraform AWS module version 4.1.0 and terraform AWS EKS module version 18.8.1. In this cluster, I have installed the ingress-nginx via helm chart version 4.0.18 which means version 1.1.2 of the ingress-nginx controller.

The cluster installation was totatlly default, and the ingress-nginx installation was also totally default. In this conditions I was expecting to work at start, no reason for extra configurations or manual adjustments on AWS Console.

With further investigation, I realized that the default Security Groups created by the terraform AWS EKS module were too restrictive. After I add the rules to allow node-to-node communications, the health checks immediately started to show all instances as healthy.

Other people commented that saw the ingress-nginx controller running on only one node. And as far I can tell, this is a expected and normal behavior. Maybe at a super large scale, more controllers may be needed, but this is not related to the correct functioning of the ingress-nginx. What happens is that when a request reaches a node, it is always redirected to the ingress-nginx controller through the kubernetes internal network. This is why is importante to verify the network security rules and ensure the node-to-node communication is available.

I was running into this issue, and from what it seems, the load balancer will run a health check at the NodePort on the ingress-nginx-controller at the http:80 destination, but for some reason that only seems to run/live on one node.

It turns out (for me) a HealthCheck NodePort is created, and works, but the load balancer will need to be manually updated run health checks on that. Example output:

➜ ~ kubectl -n ingress-nginx describe svc ingress-nginx-controller
Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
# [blah blah blah..]
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31869/TCP # <= LB Health check targets this
Endpoints:                10.244.3.183:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31015/TCP
Endpoints:                10.244.3.183:443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     31387 # <= LB Health check should target that

I also did not have this issue with the marketplace/one-click app version on the nginx ingress controller.

I just ended up using the the nginx ingress from the marketplace instead and it worked fine