Question

Pod unable to curl loadbalancer

Posted January 13, 2020 303 views
Kubernetes

Hey All,

My pods are unable to curl any domains attached to or the IP of my loadbalancer. It just times out.

It does work from an external address however. This is currently preventing cert-manager from creating a certificate because it needs to curl the domain to test it (http-01 challenge method).

I have done hours of research on this and I have come up with nothing. I don’t understand why a pod would be timing out.

Does anyone have any suggestions? If I had to guess it involves Kubernetes DNS stuff, but I have a very poor understanding of how it all works.

The exact error in the challenge is:

Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://my.domain.dev/.well-known/acme-challenge/<challenge_url>': Get http://my.domain.dev/.well-known/acme-challenge/<challenge_url>: dial tcp <load balancer ip>:80: connect: connection timed out

Thanks,
-Jacob

Add a comment
cooljacob204
By:
cooljacob204
Add a comment

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer
jkwiatkoski January 14, 2020

Hi there!

Thank you for cntacting DigitalOcean! There is a known issue in the kubernetes project(https://github.com/kubernetes/kubernetes/issues/66607)for connections from within the cluster accessing public urls to the cluster. When traffic that originates from a Kubernetes pod goes through the Load Balancer, the kube-proxy service intercepts the traffic and directs it to the service instead of letting it go out to the Load Balancer and back to the node. This can cause a variety of issues depending on the service that receives these requests. In DOKS this often results in: “timeout”, “bad header”, or SSL error messages.

Current options for workaround are the following:

Access DOKS services through their resolvable service names or by using the desired services clusterIP.

or using this service annotation on newer(1.14+) clusters:

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/2b8677c5c9f5a32a1ebe7b92cbd2b9687fee6eaa/docs/controllers/services/annotations.md#servicebetakubernetesiodo-loadbalancer-hostname

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/2b8677c5c9f5a32a1ebe7b92cbd2b9687fee6eaa/docs/controllers/services/examples/README.md#https-or-http2-load-balancer-with-hostname

Regards,

John Kwiatkoski
Senior Developer Support Engineer - Kubernetes

Reply Report
  • cooljacob204 January 14, 2020

    Thanks John, this did the trick!

    I did come across this in the past however I must have mistyped or not given cert-manager enough time before I reverted it. I let it sit for a few hours this time and it seems like it’s good to go.

    Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help