Question

kubernetes load balancer forwarding rules port changing randomly

Posted February 4, 2021 798 views
KubernetesDigitalOcean Managed Load Balancers

Our load balancer is set up to terminate HTTPS (443) and HTTP (80) connections both to an nginx ingress controller (internal port 30684), using the DO managed load balancer solution. The certificate is managed as well using Let’s Encrypt automatically from the control panel (no cert manager service). We followed this tutorial, minus the cert manager, as we assumed the DO managed solution would take care of that portion (and it works well aside from this issue!) - https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes

We have had a few occasions where the forwarding rules are changing at what appears to be random (maybe 3 or 4 times in the past year). It may be associated with the automated certificate renewal, as the last time it happened we also received a renewal notification email on the same day.

Here is some information from kubectl, not much but may show something. The ports here (30684/30397) are correct and expected, but the load balancer forwarding rules in the DO control panel change occasionally and must be set back to those shown here.

C:\Users\x>kubectl get svc --namespace=ingress-nginx
NAME            TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx   LoadBalancer   ###.###.###.###   ###.###.###.###   80:30684/TCP,443:30397/TCP   368d

C:\Users\x>kubectl get pods --namespace cert-manager
No resources found in cert-manager namespace.

The forwarding rules in the control panel should be:

TCP 80 -> TCP 30684
HTTPS 443 -> HTTP 30684 (with managed certificate)

When this issue occurs, we can go into the control panel and update the rules back to what they should be to resolve the issue. This results in HTTPS communication failures for our application until an admin can sign in to fix it. Is there a way to prevent this from happening without manual intervention? Do we need to switch to the cert-manager solution recommended by the tutorial?

3 comments
  • (Accidentally left this as an answer, lol. Reposting as a comment.)

    I just ran into this issue too—we suddenly started seeing 400 errors that said “HTTP request was sent to HTTPS port” and it took me two hours to find the cause. Manually changing the ports in the load balancer worked for us as well. The worrying thing is I have no idea why this happened or when it might happen again.

  • Noah - that’s exactly what we saw! Support has indicated that changes in the control panel are not guaranteed to be persisted, which lines up with our observations. Changes must be added as ‘annotations’ on the load balancer in kubernetes.

    I received links to the following, but I’m not sure which annotations need to be defined on my load balancer. Maybe someone with more experience can weigh in:

    https://www.digitalocean.com/docs/kubernetes/how-to/configure-load-balancers/
    https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md

    I do see this annotation on my balancer, so I think this is the right train of thought (running >kubectl get svc ingress-nginx –namespace=ingress-nginx -o yaml)

    service.beta.kubernetes.io/do-loadbalancer-certificate-id

  • I am having exactly the same issue. Each time a service is configured the load balancer will be recreated and the target port of the forwarding rule will be updated to a random port that is not the node port.

    So, instead of taking each time the correct node port from here

    kubectl -n ingress-nginx describe service nginx-ingress-ingress-nginx-controller
    

    I ended up disowning the load balancer so i don’t mess my setup up each time. This is done with the disown annotation.

    apiVersion: v1
    kind: Service
    metadata:
      name: my-service
      annotations:
        ...
        service.kubernetes.io/do-loadbalancer-disown: "true"
    

    Has anyone found any better solution?

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
2 answers

Hi @klazen108

Yes, that is correct. The Cloud Controller Manager(CCM) running on the control plane manages the configuration of the LB’s that are provisioned by Kubernetes. Any manual modifications on the LB, will be overwritten by the CCM.

In order to configure your LB you need to use kubernetes service annotations. For information on how to apply annotations, you can use the command:

kubectl annotate –help

Docs for DO service annotations can be found here: https://www.digitalocean.com/docs/kubernetes/how-to/configure-load-balancers/

Service annotations can also be found on our public github repository:

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md

Hope this helps!

Regards,

John Kwiatkoski
Senior Developer Support Engineer - Kubernetes

  • The annotations work fine but… None of the available load balancer annotations control the port forwarding rules. Am I missing something? Every annotation I place is reflected on the settings panel of the load balancer and that is great. But I can’t seem to find the annotations that influence the “forward ports” section of the settings panel.