Question

Load Balancer issues: 504 gateway errors and improper load balancing

I originally had a single droplet running Unbuntu 16.04, nginx unicorn, and Rails. I followed the tutorial here to setup TLS with Let’s encrypt: https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04

I then wanted to create another instance of this droplet and put a load balancer in front of the 2 droplets. To do this, I followed this advice from the load balancer docs:

All of the Droplets you use with your load balancer need to have the same SSL certificate. After your setup works with one backend server, you can create an image of the first Droplet to use to create additional instances.

I then pointed my original DNS record to the load balancer’s IP, rather than the first droplet’s IP. Note that my DNS is hosted with Google Domains, not DO, not sure if that is significant.

I have my load balancer set to SSL passthrough and round robin.

I am seeing 2 issues now:

  1. A large majority of my requests are going to the first droplet (as per tailing the nginx access logs), rather than being equally distributed between the two. I am testing this by rapidly sending requests with Postman and Swagger.

  2. For the 2nd droplet, I’m getting only 504 bad gateway errors. Specifically:

==> error.log <==
2019/08/16 00:41:49 [error] 14126#14126: *243 connect() to unix:/home/gc-portal/app/shared/unicorn.sock failed (111: Connection refused) while connecting to upstream, client: 167.71.106.89, server: my.domain.io, request: "POST /v2/recipes HTTP/1.1", upstream: "http://unix:/home/gc-portal/app/shared/unicorn.sock:/v2/recipes", host: "my.domain.io"

My nginx conf for this app looks like this. Note though, that the nginx conf and the unicorn conf for both droplets is identical, since I cloned the first one.

Nginx conf:

upstream gc-portal {
        server unix:/home/gc-portal/app/shared/unicorn.sock fail_timeout=0;
}

server {
        server_name my.domain.io;

        root /home/gc-portal/app/current/public;

        location /assets/  {
                gzip_static on; # serve pre-gzipped version
                expires 1M;
                add_header Cache-Control public;
        }

        location / {
                try_files $uri @app;
        }

        location @app {
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header Host $http_host;
                proxy_redirect off;
                proxy_pass http://gc-portal;
        }

     #   location /pubsub {
     #proxy_pass http://gc-portal/pubsub;
     #proxy_http_version 1.1;
     #proxy_set_header Upgrade websocket;
     #proxy_set_header Connection Upgrade;
     #proxy_set_header X-Real-IP $remote_addr;
     #proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     #proxy_set_header Host $http_host;
    #}

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/my.domain.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/my.domain.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = my.domain.io) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        listen 80;
        server_name my.domain.io;
    return 404; # managed by Certbot


}

Any thoughts on what the issue might be here?

Subscribe
Share

Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

@bobbyiliev

You were totally right about the IP whitelisting for the database-- things are working now as expected. Not the first time this has happened… it’s easily overlooked! Thanks for pointing out the answer hiding in plain sight.

Hello,

I think that the problem might be with your unicorn service on the second droplet. Can you make sure that unicorn is running and that the socket exists at /home/gc-portal/app/shared/unicorn.sock?

Also make sure that unicorn is set to start on boot so that if you have to restart your droplet, the unicorn service also starts.

Once the two droplets are healthy, the loadbalancing should start working as normal and equally distribute the load between the two droplets.

Hope that this helps! Let me know how it goes. Regards, Bobby