Pod unable to curl loadbalancer

Question

Hey All,

My pods are unable to curl any domains attached to or the IP of my loadbalancer. It just times out.

It does work from an external address however. This is currently preventing cert-manager from creating a certificate because it needs to curl the domain to test it (http-01 challenge method).

I have done hours of research on this and I have come up with nothing. I don’t understand why a pod would be timing out.

Does anyone have any suggestions? If I had to guess it involves Kubernetes DNS stuff, but I have a very poor understanding of how it all works.

The exact error in the challenge is:

Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://my.domain.dev/.well-known/acme-challenge/<challenge_url>': Get http://my.domain.dev/.well-known/acme-challenge/<challenge_url>: dial tcp <load balancer ip>:80: connect: connection timed out

Thanks, -Jacob

John Kwiatkoski · Accepted Answer

Hi there!

Thank you for cntacting DigitalOcean! There is a known issue in the kubernetes project(https://github.com/kubernetes/kubernetes/issues/66607)for connections from within the cluster accessing public urls to the cluster. When traffic that originates from a Kubernetes pod goes through the Load Balancer, the kube-proxy service intercepts the traffic and directs it to the service instead of letting it go out to the Load Balancer and back to the node. This can cause a variety of issues depending on the service that receives these requests. In DOKS this often results in: “timeout”, “bad header”, or SSL error messages.

Current options for workaround are the following:

Access DOKS services through their resolvable service names or by using the desired services clusterIP.

or using this service annotation on newer(1.14+) clusters:

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/2b8677c5c9f5a32a1ebe7b92cbd2b9687fee6eaa/docs/controllers/services/annotations.md#servicebetakubernetesiodo-loadbalancer-hostname

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/2b8677c5c9f5a32a1ebe7b92cbd2b9687fee6eaa/docs/controllers/services/examples/README.md#https-or-http2-load-balancer-with-hostname

Regards,

John Kwiatkoski Senior Developer Support Engineer - Kubernetes