Can't get https working with nginx & let's encrypt

July 25, 2017 4k views
Nginx Let's Encrypt Ubuntu
djeyewater
By:
djeyewater

I can’t access my sites over https. Exact error message is browser specific, for example “A connection was reset (corresponding to a TCP RST).”
When I try nothing is recorded in the nginx logs for the sites I’m trying to access, so I’m not sure if the request is getting as far as nginx.

Checking the server is listening on 443, running sudo netstat -plutn shows

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      31216/nginx: master

Checking if the firewall is blocking 443, running iptables -L -n -v, the only lines mentioning 443 or 80 are:

29235 1687K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80
  253 13652 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443

And checking from another machine if port 443 is open, running nc -zv mysite.com 443 shows Connection to mysite.com port [tcp/https] succeeded!

Probably the most basic server block I have in my nginx config file is:

server {
        listen          443 ssl http2;
        server_name     static2.mysite.com;

        #Allow access for lets encrypt challenge
        location /.well-known {
                allow all;
        }

        ssl_certificate /etc/letsencrypt/live/mysite.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/mysite.com/privkey.pem;
        include ssl.conf;


#rewrite to static1.
return 301 https://static1.mysite.com$request_uri;

}

(though this particular one doesn’t have logging, other server blocks that also don’t work, do have logging).

and ssl.conf looks like:

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_ciphers "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:$
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_stapling on;
ssl_stapling_verify on;
add_header Strict-Transport-Security max-age=15768000;

And I definitely restarted nginx.

So I’m not sure what the problem is, unless it’s the actual cert files generated by lets encrypt are invalid, but I don’t know how I’d check that?

Log In to Comment
2 Answers
hansen July 25, 2017

Hi @djeyewater

I’m not sure if it’s just a console cut-off, but can you change your ssl_ciphers to this:

ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';

Then restart Nginx and since you have chosen not to display your domain, can you run the SSL Labs test:
https://www.ssllabs.com/ssltest/

You can use this generator to make other cipher suites:
https://mozilla.github.io/server-side-tls/ssl-config-generator/

And since you’re using Ubuntu, I would recommend these commands for quicker overview:

sudo lsof -iTCP -sTCP:LISTEN -P
sudo ufw status
Reply
  • djeyewater July 26, 2017

    Thanks for the suggestion, the ciphers line was actually:

    ssl_ciphers "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA";

    I tried replacing it with your suggestion and restarting nginx, but still get the same result. Running the ssl Labs test gives the result Assessment failed: No secure protocols supported

    sudo lsof -iTCP -sTCP:LISTEN -P gives:

    COMMAND     PID            USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
nginx       904            root   43u  IPv4 1110584      0t0  TCP *:80 (LISTEN)
nginx       904            root   44u  IPv4 1110585      0t0  TCP *:443 (LISTEN)
nginx       905        www-data   43u  IPv4 1110584      0t0  TCP *:80 (LISTEN)
nginx       905        www-data   44u  IPv4 1110585      0t0  TCP *:443 (LISTEN)
nginx       906        www-data   43u  IPv4 1110584      0t0  TCP *:80 (LISTEN)
nginx       906        www-data   44u  IPv4 1110585      0t0  TCP *:443 (LISTEN)

    and sudo ufw status gives:

    Status: active

To                         Action      From
--                         ------      ----
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
80/tcp (v6)                ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)
    • hansen July 26, 2017

      @djeyewater
      Okay, I’ve never seen that problem before. Can you run this.

      nginx -V

      Also, your error log should contain traces of problems too, run this:

      tail -20 /var/log/nginx/error.log
      • djeyewater July 26, 2017

        Yeah, I can’t see where the problem is. nginx -V shows:

        nginx version: nginx/1.11.3
built by gcc 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2)
built with OpenSSL 1.0.2g  1 Mar 2016
TLS SNI support enabled
configure arguments: --prefix=/opt/nginx-1.11.3 --with-pcre=/tmp/tarballs/pcre-8.41 --with-pcre-jit --without-http_autoindex_module --without-http_empty_gif_module --without-http_ssi_module --with-http_ssl_module --with-http_v2_module --with-ipv6 --with-http_gzip_static_module --with-http_realip_module --with-http_geoip_module --with-http_perl_module --with-perl_modules_path=perl/lib --with-cc-opt=-I/opt/GeoIP/include --with-ld-opt='-Wl,-R,/opt/GeoIP/lib -L/opt/GeoIP/lib'

        The main error log only contains 1 error from a few days ago (when I made a typo in where the certificates were stored and so nginx wouldn’t start).

        When I tail -f the specific error and access logs for one of the sites, accessing the site on https doesn’t show anything in the logs (as if the request isn’t reaching nginx). Whereas accessing on http does show on the access log.

        The error log level is set to warn, so not sure if that has any effect on an error not being shown? I would have thought the access log would log the request though if it hit nginx, even if it was an error?

        • hansen July 27, 2017

          @djeyewater
          I’ve never seen this problem before. I simply don’t know what’s wrong with your installation, so I would recommend that you reinstall.

          • djeyewater July 30, 2017

            I created a new droplet, provisioned it, and still the same thing.
            I see there’s a new version of nginx and I think maybe it works with openssl 1.1, so I’ll try with the latest nginx and openssl and see if I get any further.

hansen July 31, 2017

@djeyewater

Even Nginx 1.11.0 should work with OpenSSL 1.1, but there has been some bug-fixes in newer versions of Nginx, so I would recommend Nginx 1.12 or higher if you want to use OpenSSL 1.1

Unless you’re compiling Nginx yourself, then there’s not a large repository, which is compiled against OpenSSL 1.1, but the Ondrej works even though it’s compiled against 1.0.2:
https://launchpad.net/~ondrej/+archive/ubuntu/nginx

I have never seen the error you’ve encountered, so I’m interested in what your end-solution to this problem is.

Reply
  • djeyewater August 8, 2017

    OK, so I have been trying to get this working since my last message. Absolutely nothing I tried worked, neither on a droplet or on a local VM (with self signed certs).

    I had an older VM I had deployed quite some time back, configured for the same sites, which worked fine. So eventually I tried copying the nginx config from this VM to a freshly deployed VM where the sites couldn’t be accessed via SSL. This worked, and I could now access the sites via SSL, pointing to a problem with the nginx config.

    However, when I diffed the two configs the only difference was the old (working) config had .well-known location blocks in the wrong places for lets encrypt to work.

    I then went through quite a lot more deployments, trying to get my deployment script back to the state I actually wanted to deploy (as I had made quite a few changes trying older versions of stuff and different settings trying to fix this problem). Sometimes https worked on these, sometimes not.

    When I got everything correct as I wanted, I deployed to a new droplet. But https didn’t work again! I then rebuilt and deployed again, but this time using self signed certs as I was using on my local VM deployments. This worked, so I then generated the cert via lets encrypt, and it still worked!

    So I have no idea what the problem was, other than it must be something to do with the nginx config. But as I said, there weren’t any errors with the non-working config, and there doesn’t appear to be any difference between a non-working and working config. But at least it looks like it’s working now for me.

Have another answer? Share your knowledge.

Related Questions