Question

Pod unable to curl loadbalancer

Hey All,

My pods are unable to curl any domains attached to or the IP of my loadbalancer. It just times out.

It does work from an external address however. This is currently preventing cert-manager from creating a certificate because it needs to curl the domain to test it (http-01 challenge method).

I have done hours of research on this and I have come up with nothing. I don’t understand why a pod would be timing out.

Does anyone have any suggestions? If I had to guess it involves Kubernetes DNS stuff, but I have a very poor understanding of how it all works.

The exact error in the challenge is:

Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://my.domain.dev/.well-known/acme-challenge/<challenge_url>': Get http://my.domain.dev/.well-known/acme-challenge/<challenge_url>: dial tcp <load balancer ip>:80: connect: connection timed out

Thanks, -Jacob


Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

John Kwiatkoski
DigitalOcean Employee
DigitalOcean Employee badge
January 14, 2020
Accepted Answer

Hi there!

Thank you for cntacting DigitalOcean! There is a known issue in the kubernetes project(https://github.com/kubernetes/kubernetes/issues/66607)for connections from within the cluster accessing public urls to the cluster. When traffic that originates from a Kubernetes pod goes through the Load Balancer, the kube-proxy service intercepts the traffic and directs it to the service instead of letting it go out to the Load Balancer and back to the node. This can cause a variety of issues depending on the service that receives these requests. In DOKS this often results in: “timeout”, “bad header”, or SSL error messages.

Current options for workaround are the following:

Access DOKS services through their resolvable service names or by using the desired services clusterIP.

or using this service annotation on newer(1.14+) clusters:

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/2b8677c5c9f5a32a1ebe7b92cbd2b9687fee6eaa/docs/controllers/services/annotations.md#servicebetakubernetesiodo-loadbalancer-hostname

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/2b8677c5c9f5a32a1ebe7b92cbd2b9687fee6eaa/docs/controllers/services/examples/README.md#https-or-http2-load-balancer-with-hostname

Regards,

John Kwiatkoski Senior Developer Support Engineer - Kubernetes

Become a contributor for community

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

DigitalOcean Documentation

Full documentation for every DigitalOcean product.

Resources for startups and SMBs

The Wave has everything you need to know about building a business, from raising funding to marketing your product.

Get our newsletter

Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.

New accounts only. By submitting your email you agree to our Privacy Policy

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Get started for free

Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

*This promotional offer applies to new accounts only.