Question

How do I correct a "connection timed out" error during http-01 challenge propagation with Cert-Manager?

Posted December 28, 2019 33.7k views
NginxLet's EncryptKubernetes

I’m following the How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes tutorial to try and set up HTTPS ingresses for my cluster.

I’ve got it working up until the end. However, the certificates never get issued. If I drill down from the Certificate, to the CertificateRequest, to the Order, to the Challenge, I eventually find this errorr:

Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://cert-test.tanndev.com/.well-known/acme-challenge/kC9hDBr8qI89
8y1gqacU2BbytGBb21YfQHaCyUx1kEY': Get http://cert-test.tanndev.com/.well-known/acme-challenge/kC9hDBr8qI898y1gqacU2BbytGBb21YfQHaCyUx1kEY: dial tcp 165.227.252.80:80: c
onnect: connection timed out

Clearly, this is blocking the certificate from being issued. But I’m not sure what’s causing it because I can reach that url just fine, via both a browser and CURL.

I’m not sure what else to even look for, so any suggestions would be greatly appreciated.

If it helps, the manifest for the cert-test app is here, and the manifest for the certificate issuer is here.

Thanks in advance!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
3 answers

I found a fix for the issue thanks to the kind soul at this link; who points to this official digital ocean answer/workaround.

The issue is with an incompatibily with DO’s loadBalancer and the way k8s works.

The workaround is:

  1. create a dedicated DNS entry pointing to the public IP of your load balancer (checking for EXTERNAL-IP via `kubectl get svc -n ingress-nginx), eg. kube.mydomain.com

  2. In the manifest used to create your load balancer (probably this, add an annotation pointing ot that newly created DNS entry, like described in the DO doc.
    eg.

annotations: 
    service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"

Here is my full load balancer manifest for reference:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  annotations: 
    # See https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/examples/README.md#accessing-pods-over-a-managed-load-balancer-from-inside-the-cluster
    service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https


This fixed the issue for me!
Hope it helps.

Does anyone have a more detailed explanation for how to resolve this? The links provided to the issues aren’t really clear on what needs to be changed. I would have expected the DigitalOcean’s tutorials to be updated to properly reflect a working example…

Is someone from the DO team able to weigh in on getting this to work with their cloud service?