Question

How do I correct a "connection timed out" error during http-01 challenge propagation with Cert-Manager?

Posted December 28, 2019 26.3k views
NginxLet's EncryptKubernetes

I’m following the How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes tutorial to try and set up HTTPS ingresses for my cluster.

I’ve got it working up until the end. However, the certificates never get issued. If I drill down from the Certificate, to the CertificateRequest, to the Order, to the Challenge, I eventually find this errorr:

Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://cert-test.tanndev.com/.well-known/acme-challenge/kC9hDBr8qI89
8y1gqacU2BbytGBb21YfQHaCyUx1kEY': Get http://cert-test.tanndev.com/.well-known/acme-challenge/kC9hDBr8qI898y1gqacU2BbytGBb21YfQHaCyUx1kEY: dial tcp 165.227.252.80:80: c
onnect: connection timed out

Clearly, this is blocking the certificate from being issued. But I’m not sure what’s causing it because I can reach that url just fine, via both a browser and CURL.

I’m not sure what else to even look for, so any suggestions would be greatly appreciated.

If it helps, the manifest for the cert-test app is here, and the manifest for the certificate issuer is here.

Thanks in advance!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
3 answers

I found a fix for the issue thanks to the kind soul at this link; who points to this official digital ocean answer/workaround.

The issue is with an incompatibily with DO’s loadBalancer and the way k8s works.

The workaround is:

  1. create a dedicated DNS entry pointing to the public IP of your load balancer (checking for EXTERNAL-IP via `kubectl get svc -n ingress-nginx), eg. kube.mydomain.com

  2. In the manifest used to create your load balancer (probably this, add an annotation pointing ot that newly created DNS entry, like described in the DO doc.
    eg.

annotations: 
    service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"

Here is my full load balancer manifest for reference:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  annotations: 
    # See https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/examples/README.md#accessing-pods-over-a-managed-load-balancer-from-inside-the-cluster
    service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https


This fixed the issue for me!
Hope it helps.

  • Thank you for sharing.

    For those who followed the official tutorial, paste and apply the file above instead of doing this step kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml.

    by Hanif Jetha
    In this tutorial, learn how to set up and secure an Nginx Ingress Controller with Cert-Manager on DigitalOcean Kubernetes.
  • Dude you are the man! I was going insane with this as it worked 2 months ago and suddenly I wasn’t able to issue certs. Guessing this means Digital Ocean has changed something with their LBs recently to suddenly break this?

    • I dont think this is right, “kube.mydomain.com” ???

      Where do you get this “kube.mydomain.com” from, shouldnt that actually be the actual load balancer host-name? or can it actually just be this generic “kube.mydomain.com”???

      • It did the trick for me and others here.
        Read my initial comment again, and I think you will understand where kube.mydomain.com comes from.

        Quoting from that comment:

        1. create a dedicated DNS entry pointing to the public IP of your load balancer (checking for EXTERNAL-IP via `kubectl get svc -n ingress-nginx), eg. kube.mydomain.com
  • Indeed, this fixed it for me too. Thanks.

Does anyone have a more detailed explanation for how to resolve this? The links provided to the issues aren’t really clear on what needs to be changed. I would have expected the DigitalOcean’s tutorials to be updated to properly reflect a working example…

Is someone from the DO team able to weigh in on getting this to work with their cloud service?

Submit an Answer