Question

Loadbalancer reports droplet with express server and Nginx Reverse Proxy as down

Posted July 13, 2021 101 views
NginxNode.jsLoad Balancing

Hi there,

The setup for my network is as follows:

  • A DO load balancer with SSL termination, as explained here
  • A DO droplet with an Nginx reverse proxy, using the config generated by the DO NGINXConfig tool, found here

Just to note also, I have enabled Proxy Protocol on my load balancer and passed the required headers where necessary.

For some reason though, my load balancer is saying my droplet is down when its not.

I assume I’m missing something but was wondering if anyone has any idea as to what.

Thanks!

1 comment
  • After spending more time trying to work out what’s causing this, I believe it’s something to do with the fact that my droplet is using HTTP and the frontend is HTTPS.

    Although the loadbalancer is terminating the SSL request, I do not think the security headers created by the NGINXConfig tool support this.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
1 answer

Hi there,

What I could suggest is changing the health check from HTTP to TCP.

DigitalOcean Loadbalancer TCP config

Also, make sure that the check port matches the one that your backend service is running on.

Let me know how it goes!
Regards,
Bobby

  • Hi @bobbyiliev,

    Thanks very much for the reply. This worked, however i’m not sure why, plus, it has come at a compromise:

    Previously to the setup I describe above, I was using the exact same setup, only with HTTPS rather than HTTP between my load balancer and NGinx reverse proxy on the droplet. Using this previous setup, the HTTP health check worked fine, so I don’t know why this change would break it because the express server themselves are no different.

    The compromise I talk about also is that if my express server goes down, the healthcheck still returns healthy, because it’s only pinging the port, meaning traffic is sent there still. Previously, if my express server went down, the health check would fail and no traffic would be sent to that droplet.

    I am unsure why the change from HTTPS to HTTP between my load balancer and NGinx reverse proxy on my droplet caused the healthcheck to fail via HTTP. I’m not sure if you have any ideas?

    • Hi there,

      What is the output of the following command:

      netstat -plant | grep '80\|443'
      

      Also if you try to do a curl -IL your_droplet_ip does it work with both HTTP and HTTPS?

      Regards,
      Bobby

      • Hey,

        What I think is relevant from the output of the first command is:

        tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      21170/nginx: master 
        tcp6       0      0 :::80                   :::*                    LISTEN      21170/nginx: master 
        

        I can dump the whole thing (it had a few other ports and ips listed)

        Running the second command, both timed out - however this is expected because my droplet is behind a firewall, which only allows the load balancer to connect.

        • Hi there,

          If you allow your IP via the firewall and then try to do a curl -IL droplet_ip do you see a 200 OK status in the headers?

          I think that it is possible that your default Nginx server block could be different from your server block for your domain which could have a different response when reaching it.

          • When I remove my droplet from the firewall, and run the command you said, the message is:

            curl: (52) Empty reply from server

            I doubled checked and port 80 is open when I do this.

          • Hello,

            This sounds like that this could be the reason why the load balancer health checks are failing.

            Do you see any errors in the Nginx error log? Also, are you using multiple Nginx server blocks which could be causing a conflict?

            Feel free to share them here after removing any sensitive information.

          • Hi Bobby,

            Apologies, for some reason I can’t reply to your latest message so Im replying to this one.

            I see the following error in my nginx error log:

            ^@^H^@^]^@^W^@^Y^@^X^@#^@^@^@^V^@^@^@^W^@^@^@^M^@ ^@^^^F^A^F^B^F^C^E^A^E^B^E^C^D^A^D^B^D^C^C^A^C^B^C^C^B^A^B^B^B^C" while reading PROXY protocol, client: IP_REMOVED, server: 0.>
            2021/07/26 11:17:46 [error] 693#693: *70221 broken header: "�^@^@(>�L1^@^@^@^@^@^@^@^B^@^A��^@^@^@^B^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@" while reading PROXY protocol, client: >
            
            

            The only Nginx server block I use is for my api on localhost:4000:

            server {
                listen      80 proxy_protocol;
                listen      [::]:80 proxy_protocol;
                server_name api.mydomain.net;
            
                # security
                include     nginxconfig.io/security.conf;
            
                # reverse proxy
                location / {
                    proxy_pass http://localhost:4000;
                    include    nginxconfig.io/proxy.conf;
                }
            
                # additional config
                include nginxconfig.io/general.conf;
            }
            
            
          • Hey Alex,

            Ok on some further investigation, I’ve come across this Nginx bug report here:

            https://trac.nginx.org/nginx/ticket/886

            This indicates that the proxy_protocol declaration in your Nginx server block might be causing the problem.

            I could suggest removing it as a test to see if it fixes the problem.

            As far as I can see, people are reporting that everything worked as expected with Nginx v1.8.0, but when the same problem occurred with v1.8.1 which might explain your situation too.

            Let me know how this goes!

          • Hi Bobby,

            Again sorry, I can’t reply to your latest reply.

            Thanks very much for the info re proxy_protocol.

            I guess this is a limitation of my setup then? If I want to be behind my loadbalancer, and have the client IP passed, I need to enable the proxy_protocol afaik? Therefore I cant remove the proxy_protocol and have to accept this limitation?

          • Hi there,

            I think that we’ve reached the limit of nested comments.

            I think that I went into the wrong direction with that curl error, so I’ve done some further testing, I’ve set up a fresh new load balancer with SSL termination + a new Droplet running the latest Nginx version, I replicated your exact setup including the Nginx config from the Nginx tool.

            However, with this exact configuration, the health check worked as expected.

            What I could suggest is to check your Nginx access logs and see if you get any references for HTTP requests from the health check:

            tail -f /var/log/nginx/access.log
            

            In my case I’m seeing the requests each few seconds:

            - - [26/Jul/2021:17:54:18 +0000] "GET / HTTP/1.0" 200 45 "-" "-"
            - - [26/Jul/2021:17:54:21 +0000] "GET / HTTP/1.0" 200 45 "-" "-"
            - - [26/Jul/2021:17:54:24 +0000] "GET / HTTP/1.0" 200 45 "-" "-"
            - - [26/Jul/2021:17:54:27 +0000] "GET / HTTP/1.0" 200 45 "-" "-"
            - - [26/Jul/2021:17:54:30 +0000] "GET / HTTP/1.0" 200 45 "-" "-"
            - - [26/Jul/2021:17:54:33 +0000] "GET / HTTP/1.0" 200 45 "-" "-"
            - - [26/Jul/2021:17:54:37 +0000] "GET / HTTP/1.0" 200 45 "-" "-"
            - - [26/Jul/2021:17:54:40 +0000] "GET / HTTP/1.0" 200 45 "-" "-"
            - - [26/Jul/2021:17:54:43 +0000] "GET / HTTP/1.0" 200 45 "-" "-"
            - - [26/Jul/2021:17:54:46 +0000] "GET / HTTP/1.0" 200 45 "-" "-"
            

            Also you could try the same with the error log to see if you get some more information there:

            tail -f /var/log/nginx/error.log
            

            Regards,
            Bobby

          • Hi Bobby,

            Thanks very much indeed for taking the time to do that. Answering your question, when I look at the info inside access.log I see a few different things:

            When I connect “correctly” via my website url, I see the following:

            10.106.0.7 - - [27/Jul/2021:05:41:17 +0000] "POST / HTTP/1.1" 200 1096 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
            

            However, if I navigate directly to the droplet IP, or go to api.mydomain.net in the browser, I see the following:

            10.106.0.7 - - [27/Jul/2021:04:32:25 +0000] "GET / HTTP/1.1" 500 148 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
            

            When I look inside error.log I see a large number of issues, I am unsure whats relevant, so here is a dump:

            2021/07/26 11:17:16 [error] 693#693: *70153 broken header: "HEAD / HTTP/1.1
            Host: 165.227.227.145
            User-Agent: curl/7.64.1
            Accept: */*
            
            " while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:44 [error] 693#693: *70157 broken header: "GET / HTTP/1.0
            
            " while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:44 [error] 693#693: *70158 broken header: "OPTIONS / HTTP/1.0
            
            " while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:44 [error] 693#693: *70159 broken header: "OPTIONS / RTSP/1.0
            
            " while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:44 [error] 693#693: *70160 broken header: "l^@^K^@^@^@^@^@^@^@^@^@" while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:44 [error] 693#693: *70161 broken header: "GET /nice%20ports%2C/Tri%6Eity.txt%2ebak HTTP/1.0
            
            " while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:44 [error] 693#693: *70162 broken header: "
            
            " while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:44 [error] 693#693: *70163 broken header: "�^@^@(r�^]^S^@^@^@^@^@^@^@^B^@^A��^@^A�|^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@" while reading PROXY protocol, client: REMOVED_IP, server: 0.>
            2021/07/26 11:17:45 [error] 693#693: *70164 broken header: "^@^^^@^F^A^@^@^A^@^@^@^@^@^@^Gversion^Dbind^@^@^P^@^C" while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:45 [error] 693#693: *70165 broken header: "^@^L^@^@^P^@^@^@^@^@^@^@^@^@" while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:45 [error] 693#693: *70166 broken header: "HELP
            " while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:45 [error] 693#693: *70167 broken header: "^V^C^@^@S^A^@^@O^C^@?G���,���`~�^@��{�Ֆ�w����<=�o�^Pn^@^@(^@^V^@^S^@
            ^@f^@^E^@^D^@e^@d^@c^@b^@a^@`^@^U^@^R^@ ^@^T^@^Q^@^H^@^F^@^C^A^@" while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:45 [error] 693#693: *70168 broken header: "^V^C^@^@i^A^@^@e^C^CU^\��random1random2random3random4^@^@^L^@/^@
            ^@^S^@9^@^D^@�^A^@^@0^@^M^@,^@*^@^A^@^C^@^B^F^A^F^C^F^B^B^A^B^C^B^B^C^A^C^C^C^B^D^A^D^C^D^B^A^A^A^C^A^B^E^A^E^C^E^B" while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80
            2021/07/26 11:17:45 [error] 693#693: *70169 broken header: "^@^@^@qj�n0�k�^C^B^A^E�^C^B^A
            ��^0\�^G^C^E^@P�^@^P�^D^[^BNM�^W0^U�^C^B^A^@�^N0^L^[^Fkrbtgt^[^BNM�^Q^X^O19700101000000Z�^F^B^D^_^^�٨^W0^U^B^A^R^B^A^Q^B^A^P^B^A^W^B^A^A^B^A^C^B^A^B" while reading PROXY protocol, client: REMOVED_IP, se>
            2021/07/26 11:17:45 [error] 693#693: *70170 broken header: "^@^@^@��SMBr^@^@^@^@^H^A@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@^F^@^@^A^@^@�^@^BPC NETWORK PROGRAM 1.0^@^BMICROSOFT NETWORKS 1.03^@^BMICROSOFT NETWORKS 3.0>
            2021/07/26 11:17:45 [error] 693#693: *70171 broken header: "^Adefault
            " while reading PROXY protocol, client: REMOVED_IP, server: 0.0.0.0:80