Question

kubernetes load balancer forwarding rules port changing randomly

Our load balancer is set up to terminate HTTPS (443) and HTTP (80) connections both to an nginx ingress controller (internal port 30684), using the DO managed load balancer solution. The certificate is managed as well using Let’s Encrypt automatically from the control panel (no cert manager service). We followed this tutorial, minus the cert manager, as we assumed the DO managed solution would take care of that portion (and it works well aside from this issue!) - https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes

We have had a few occasions where the forwarding rules are changing at what appears to be random (maybe 3 or 4 times in the past year). It may be associated with the automated certificate renewal, as the last time it happened we also received a renewal notification email on the same day.

Here is some information from kubectl, not much but may show something. The ports here (30684/30397) are correct and expected, but the load balancer forwarding rules in the DO control panel change occasionally and must be set back to those shown here.

C:\Users\x>kubectl get svc --namespace=ingress-nginx
NAME            TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx   LoadBalancer   ###.###.###.###   ###.###.###.###   80:30684/TCP,443:30397/TCP   368d

C:\Users\x>kubectl get pods --namespace cert-manager
No resources found in cert-manager namespace.

The forwarding rules in the control panel should be:

TCP 80 -> TCP 30684 HTTPS 443 -> HTTP 30684 (with managed certificate)

When this issue occurs, we can go into the control panel and update the rules back to what they should be to resolve the issue. This results in HTTPS communication failures for our application until an admin can sign in to fix it. Is there a way to prevent this from happening without manual intervention? Do we need to switch to the cert-manager solution recommended by the tutorial?

Show comments

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

Hi @klazen108

Yes, that is correct. The Cloud Controller Manager(CCM) running on the control plane manages the configuration of the LB’s that are provisioned by Kubernetes. Any manual modifications on the LB, will be overwritten by the CCM.

In order to configure your LB you need to use kubernetes service annotations. For information on how to apply annotations, you can use the command:

kubectl annotate --help

Docs for DO service annotations can be found here: https://www.digitalocean.com/docs/kubernetes/how-to/configure-load-balancers/

Service annotations can also be found on our public github repository:

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md

Hope this helps!

Regards,

John Kwiatkoski Senior Developer Support Engineer - Kubernetes

This comment has been deleted