Related

Product Managed Kubernetes on DigitalOcean Product
NGINX Config Generator Tool
Certbot nginx plugin could not open file Question Question
how to fix Leverage browser caching nginx? Question Question

Question

How do I correct a "connection timed out" error during http-01 challenge propagation with Cert-Manager?

Posted December 28, 2019 22.7k views
NginxLet's EncryptKubernetes

I’m following the How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes tutorial to try and set up HTTPS ingresses for my cluster.

I’ve got it working up until the end. However, the certificates never get issued. If I drill down from the Certificate, to the CertificateRequest, to the Order, to the Challenge, I eventually find this errorr:

Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://cert-test.tanndev.com/.well-known/acme-challenge/kC9hDBr8qI89
8y1gqacU2BbytGBb21YfQHaCyUx1kEY': Get http://cert-test.tanndev.com/.well-known/acme-challenge/kC9hDBr8qI898y1gqacU2BbytGBb21YfQHaCyUx1kEY: dial tcp 165.227.252.80:80: c
onnect: connection timed out

Clearly, this is blocking the certificate from being issued. But I’m not sure what’s causing it because I can reach that url just fine, via both a browser and CURL.

I’m not sure what else to even look for, so any suggestions would be greatly appreciated.

If it helps, the manifest for the cert-test app is here, and the manifest for the certificate issuer is here.

Thanks in advance!

Add a comment
jftanner
By:
jftanner
Add a comment

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
3 answers
dr42 January 7, 2020

I found a fix for the issue thanks to the kind soul at this link; who points to this official digital ocean answer/workaround.

The issue is with an incompatibily with DO’s loadBalancer and the way k8s works.

The workaround is:

  1. create a dedicated DNS entry pointing to the public IP of your load balancer (checking for EXTERNAL-IP via `kubectl get svc -n ingress-nginx), eg. kube.mydomain.com

  2. In the manifest used to create your load balancer (probably this, add an annotation pointing ot that newly created DNS entry, like described in the DO doc.
    eg. 

annotations: 
    service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"

Here is my full load balancer manifest for reference:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  annotations: 
    # See https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/examples/README.md#accessing-pods-over-a-managed-load-balancer-from-inside-the-cluster
    service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

This fixed the issue for me!
Hope it helps.

Reply Report
  • Tainan January 13, 2020

    Thank you for sharing.

    For those who followed the official tutorial, paste and apply the file above instead of doing this step kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml.

    How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes
    by Hanif Jetha
    In this tutorial, learn how to set up and secure an Nginx Ingress Controller with Cert-Manager on DigitalOcean Kubernetes.
    Reply Report
  • maelstrome26 February 16, 2020

    Dude you are the man! I was going insane with this as it worked 2 months ago and suddenly I wasn’t able to issue certs. Guessing this means Digital Ocean has changed something with their LBs recently to suddenly break this?

    Reply Report
    • nigeldewar March 5, 2020

      I dont think this is right, “kube.mydomain.com” ???

      Where do you get this “kube.mydomain.com” from, shouldnt that actually be the actual load balancer host-name? or can it actually just be this generic “kube.mydomain.com”???

      Reply Report
      • dr42 March 8, 2020

        It did the trick for me and others here.
        Read my initial comment again, and I think you will understand where kube.mydomain.com comes from.

        Quoting from that comment:

        1. create a dedicated DNS entry pointing to the public IP of your load balancer (checking for EXTERNAL-IP via `kubectl get svc -n ingress-nginx), eg. kube.mydomain.com
        Reply Report
  • fabiansturm February 28, 2020

    Thank you!

    Reply Report
  • Codebard March 8, 2020

    Indeed, this fixed it for me too. Thanks.

    Reply Report
jballeaume December 30, 2019
Reply Report
vanpyrzericj January 3, 2020

Does anyone have a more detailed explanation for how to resolve this? The links provided to the issues aren’t really clear on what needs to be changed. I would have expected the DigitalOcean’s tutorials to be updated to properly reflect a working example…

Is someone from the DO team able to weigh in on getting this to work with their cloud service?

Reply Report
  • jftanner January 3, 2020

    If found the answer (or, perhaps just a workaround) while waiting for admin approval on this question: I set externalTrafficPolicy to Cluster in my LoadBalancer, and then it worked.

    My assumption is that is that my pod with the certificate-issuer wound up on a different node than the load balancer did, so it couldn’t talk to itself through the ingress. I could be totally wrong about that, but the setting worked at least.

    Reply Report
    • nathan8b799cb7e February 17, 2020

      Making this one change on the externalTrafficPolicy did the trick for me.

      So instead of this:
      kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml

      Apply this instead:

      kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  **externalTrafficPolicy: Cluster**
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https
      Reply Report
      • kenmoini May 6, 2020

        thisthisthisthisthis. For the longest time I thought it was cert-manager being garbage, but it’s actually DO LB service manifest…wow. Thanks!

        Reply Report
      • joecreager June 10, 2020

        Setting externalTrafficPolicy: Cluster was the solution for me as well. I serve multiple domains from my cluster and I was concerned about the implications of setting the host name annotation on the load balancer to a single host. Glad this did the trick.

        Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help