Deploying ingress-nginx on new cluster creates load balancer that fails health check

Deploying ingress-nginx on new cluster creates load balancer that fails health check When I deploy ingress-nginx, a load balancer is created that points to two nodes. One healthy and one down.

I had the same issue when deploying a brand new kubernetes with AWS EKS using terraform AWS module version 4.1.0 and terraform AWS EKS module version 18.8.1. In this cluster, I have installed the ingress-nginx via helm chart version 4.0.18 which means version 1.1.2 of the ingress-nginx controller.

The cluster installation was totatlly default, and the ingress-nginx installation was also totally default. In this conditions I was expecting to work at start, no reason for extra configurations or manual adjustments on AWS Console.

With further investigation, I realized that the default Security Groups created by the terraform AWS EKS module were too restrictive. After I add the rules to allow node-to-node communications, the health checks immediately started to show all instances as healthy.

Other people commented that saw the ingress-nginx controller running on only one node. And as far I can tell, this is a expected and normal behavior. Maybe at a super large scale, more controllers may be needed, but this is not related to the correct functioning of the ingress-nginx. What happens is that when a request reaches a node, it is always redirected to the ingress-nginx controller through the kubernetes internal network. This is why is importante to verify the network security rules and ensure the node-to-node communication is available.

I was running into this issue, and from what it seems, the load balancer will run a health check at the NodePort on the ingress-nginx-controller at the http:80 destination, but for some reason that only seems to run/live on one node.

It turns out (for me) a HealthCheck NodePort is created, and works, but the load balancer will need to be manually updated run health checks on that. Example output:

➜ ~ kubectl -n ingress-nginx describe svc ingress-nginx-controller
Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
# [blah blah blah..]
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31869/TCP # <= LB Health check targets this
Endpoints:                10.244.3.183:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31015/TCP
Endpoints:                10.244.3.183:443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     31387 # <= LB Health check should target that

I also did not have this issue with the marketplace/one-click app version on the nginx ingress controller.

I just ended up using the the nginx ingress from the marketplace instead and it worked fine