Pod unable to curl loadbalancer

January 13, 2020 56 views
Kubernetes

Hey All,

My pods are unable to curl any domains attached to or the IP of my loadbalancer. It just times out.

It does work from an external address however. This is currently preventing cert-manager from creating a certificate because it needs to curl the domain to test it (http-01 challenge method).

I have done hours of research on this and I have come up with nothing. I don’t understand why a pod would be timing out.

Does anyone have any suggestions? If I had to guess it involves Kubernetes DNS stuff, but I have a very poor understanding of how it all works.

The exact error in the challenge is:

Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://my.domain.dev/.well-known/acme-challenge/<challenge_url>': Get http://my.domain.dev/.well-known/acme-challenge/<challenge_url>: dial tcp <load balancer ip>:80: connect: connection timed out

Thanks,
-Jacob

1 Answer

Hi there!

Thank you for cntacting DigitalOcean! There is a known issue in the kubernetes project(https://github.com/kubernetes/kubernetes/issues/66607)for connections from within the cluster accessing public urls to the cluster. When traffic that originates from a Kubernetes pod goes through the Load Balancer, the kube-proxy service intercepts the traffic and directs it to the service instead of letting it go out to the Load Balancer and back to the node. This can cause a variety of issues depending on the service that receives these requests. In DOKS this often results in: “timeout”, “bad header”, or SSL error messages.

Current options for workaround are the following:

Access DOKS services through their resolvable service names or by using the desired services clusterIP.

or using this service annotation on newer(1.14+) clusters:

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/2b8677c5c9f5a32a1ebe7b92cbd2b9687fee6eaa/docs/controllers/services/annotations.md#servicebetakubernetesiodo-loadbalancer-hostname

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/2b8677c5c9f5a32a1ebe7b92cbd2b9687fee6eaa/docs/controllers/services/examples/README.md#https-or-http2-load-balancer-with-hostname

Regards,

John Kwiatkoski
Senior Developer Support Engineer - Kubernetes

  • Thanks John, this did the trick!

    I did come across this in the past however I must have mistyped or not given cert-manager enough time before I reverted it. I let it sit for a few hours this time and it seems like it’s good to go.

Have another answer? Share your knowledge.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!