How do I correct a "connection timed out" error during http-01 challenge propagation with Cert-Manager?

Question

I’m following the How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes tutorial to try and set up HTTPS ingresses for my cluster.

I’ve got it working up until the end. However, the certificates never get issued. If I drill down from the Certificate, to the CertificateRequest, to the Order, to the Challenge, I eventually find this errorr:

Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://cert-test.tanndev.com/.well-known/acme-challenge/kC9hDBr8qI89
8y1gqacU2BbytGBb21YfQHaCyUx1kEY': Get http://cert-test.tanndev.com/.well-known/acme-challenge/kC9hDBr8qI898y1gqacU2BbytGBb21YfQHaCyUx1kEY: dial tcp 165.227.252.80:80: c
onnect: connection timed out

Clearly, this is blocking the certificate from being issued. But I’m not sure what’s causing it because I can reach that url just fine, via both a browser and CURL.

I’m not sure what else to even look for, so any suggestions would be greatly appreciated.

If it helps, the manifest for the cert-test app is here, and the manifest for the certificate issuer is here.

Thanks in advance!

Answer 1

dr42 January 7, 2020

I found a fix for the issue thanks to the kind soul at this link; who points to this official digital ocean answer/workaround.

The issue is with an incompatibily with DO’s loadBalancer and the way k8s works.

The workaround is:

create a dedicated DNS entry pointing to the public IP of your load balancer (checking for EXTERNAL-IP via `kubectl get svc -n ingress-nginx), eg. kube.mydomain.com
In the manifest used to create your load balancer (probably this, add an annotation pointing ot that newly created DNS entry, like described in the DO doc.
eg.

annotations: 
    service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"

Here is my full load balancer manifest for reference:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  annotations: 
    # See https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/examples/README.md#accessing-pods-over-a-managed-load-balancer-from-inside-the-cluster
    service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

This fixed the issue for me!
Hope it helps.

Reply Report

Tainan January 13, 2020

Thank you for sharing.

For those who followed the official tutorial, paste and apply the file above instead of doing this step kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml.

How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes
by Hanif Jetha
In this tutorial, learn how to set up and secure an Nginx Ingress Controller with Cert-Manager on DigitalOcean Kubernetes.
Reply Report
maelstrome26 February 16, 2020

Dude you are the man! I was going insane with this as it worked 2 months ago and suddenly I wasn’t able to issue certs. Guessing this means Digital Ocean has changed something with their LBs recently to suddenly break this?
Reply Report
- nigeldewar March 5, 2020
  
  I dont think this is right, “kube.mydomain.com” ???
  
  Where do you get this “kube.mydomain.com” from, shouldnt that actually be the actual load balancer host-name? or can it actually just be this generic “kube.mydomain.com”???
  
  Reply Report
  
  dr42 March 8, 2020
  
  It did the trick for me and others here.
  Read my initial comment again, and I think you will understand where kube.mydomain.com comes from.
  
  Quoting from that comment:
  
  create a dedicated DNS entry pointing to the public IP of your load balancer (checking for EXTERNAL-IP via `kubectl get svc -n ingress-nginx), eg. kube.mydomain.com
  
  Reply Report
fabiansturm February 28, 2020

Thank you!
Reply Report
Codebard March 8, 2020

Indeed, this fixed it for me too. Thanks.
Reply Report

Answer 2

Tainan January 13, 2020

Thank you for sharing.

For those who followed the official tutorial, paste and apply the file above instead of doing this step kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml.

How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes
by Hanif Jetha
In this tutorial, learn how to set up and secure an Nginx Ingress Controller with Cert-Manager on DigitalOcean Kubernetes.
Reply Report
maelstrome26 February 16, 2020

Dude you are the man! I was going insane with this as it worked 2 months ago and suddenly I wasn’t able to issue certs. Guessing this means Digital Ocean has changed something with their LBs recently to suddenly break this?
Reply Report
- nigeldewar March 5, 2020
  
  I dont think this is right, “kube.mydomain.com” ???
  
  Where do you get this “kube.mydomain.com” from, shouldnt that actually be the actual load balancer host-name? or can it actually just be this generic “kube.mydomain.com”???
  
  Reply Report
  
  dr42 March 8, 2020
  
  It did the trick for me and others here.
  Read my initial comment again, and I think you will understand where kube.mydomain.com comes from.
  
  Quoting from that comment:
  
  create a dedicated DNS entry pointing to the public IP of your load balancer (checking for EXTERNAL-IP via `kubectl get svc -n ingress-nginx), eg. kube.mydomain.com
  
  Reply Report
fabiansturm February 28, 2020

Thank you!
Reply Report
Codebard March 8, 2020

Indeed, this fixed it for me too. Thanks.
Reply Report

Answer 3

jballeaume December 30, 2019

Hi,

Have a look at this : https://github.com/kubernetes/ingress-nginx/issues/3996#issuecomment-566460867
And https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md#servicebetakubernetesiodo-loadbalancer-hostname

Reply Report

Answer 4

vanpyrzericj January 3, 2020

Does anyone have a more detailed explanation for how to resolve this? The links provided to the issues aren’t really clear on what needs to be changed. I would have expected the DigitalOcean’s tutorials to be updated to properly reflect a working example…

Is someone from the DO team able to weigh in on getting this to work with their cloud service?

Reply Report

jftanner January 3, 2020

If found the answer (or, perhaps just a workaround) while waiting for admin approval on this question: I set externalTrafficPolicy to Cluster in my LoadBalancer, and then it worked.

My assumption is that is that my pod with the certificate-issuer wound up on a different node than the load balancer did, so it couldn’t talk to itself through the ingress. I could be totally wrong about that, but the setting worked at least.
Reply Report
- nathan8b799cb7e February 17, 2020
  
  Making this one change on the externalTrafficPolicy did the trick for me.
  
  So instead of this:
  kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml
  
  Apply this instead:
  
  kind: Service apiVersion: v1 metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: **externalTrafficPolicy: Cluster** type: LoadBalancer selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx ports: - name: http port: 80 targetPort: http - name: https port: 443 targetPort: https
  
  Reply Report
  
  kenmoini May 6, 2020
  
  thisthisthisthisthis. For the longest time I thought it was cert-manager being garbage, but it’s actually DO LB service manifest…wow. Thanks!
  
  Reply Report
  
  joecreager June 10, 2020
  
  Setting externalTrafficPolicy: Cluster was the solution for me as well. I serve multiple domains from my cluster and I was concerned about the implications of setting the host name annotation on the load balancer to a single host. Glad this did the trick.
  
  Reply Report
  
  chadacious July 18, 2020
  
  Yup! Thanks so much. There is a Note that alludes to this in this DO tutorial as well: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes
  
  How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes
  by Hanif Jetha
  In this tutorial, learn how to set up and secure an Nginx Ingress Controller with Cert-Manager on DigitalOcean Kubernetes.
  
  Reply Report

Answer 5

jftanner January 3, 2020

If found the answer (or, perhaps just a workaround) while waiting for admin approval on this question: I set externalTrafficPolicy to Cluster in my LoadBalancer, and then it worked.

My assumption is that is that my pod with the certificate-issuer wound up on a different node than the load balancer did, so it couldn’t talk to itself through the ingress. I could be totally wrong about that, but the setting worked at least.
Reply Report
- nathan8b799cb7e February 17, 2020
  
  Making this one change on the externalTrafficPolicy did the trick for me.
  
  So instead of this:
  kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml
  
  Apply this instead:
  
  kind: Service apiVersion: v1 metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: **externalTrafficPolicy: Cluster** type: LoadBalancer selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx ports: - name: http port: 80 targetPort: http - name: https port: 443 targetPort: https
  
  Reply Report
  
  kenmoini May 6, 2020
  
  thisthisthisthisthis. For the longest time I thought it was cert-manager being garbage, but it’s actually DO LB service manifest…wow. Thanks!
  
  Reply Report
  
  joecreager June 10, 2020
  
  Setting externalTrafficPolicy: Cluster was the solution for me as well. I serve multiple domains from my cluster and I was concerned about the implications of setting the host name annotation on the load balancer to a single host. Glad this did the trick.
  
  Reply Report
  
  chadacious July 18, 2020
  
  Yup! Thanks so much. There is a Note that alludes to this in this DO tutorial as well: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes
  
  How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes
  by Hanif Jetha
  In this tutorial, learn how to set up and secure an Nginx Ingress Controller with Cert-Manager on DigitalOcean Kubernetes.
  
  Reply Report

Related

Question

How do I correct a "connection timed out" error during http-01 challenge propagation with Cert-Manager?

Related

Leave a comment

Looking for something else?

Sign up for our newsletter

Related

Question

How do I correct a "connection timed out" error during http-01 challenge propagation with Cert-Manager?

Related

Leave a comment

Related

question

How to configure URL extensions at Nginx

question

Dokku node app memory allocation error

question

Error while renewing Let's Encrypt

question

Inline code tag not working in Markdown on Ghost blog

Looking for something else?