Nginx 111: Connection refused errors.

October 15, 2015 67.8k views
Nginx DigitalOcean

Hello, I'm running Nginx /unicorn I'm getting errors "failed (111: Connection refused) while connecting to upstream" from nginx. It seems I've tried just about everything DO support, google and stackoverflow has to offer. I dont know if I'm overlooking something or I've tried so much that i've now messed something up. Here is all the configs, analysis, ect...

Nginx.conf

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;
worker_rlimit_nofile 30000;

events { worker_connections 1024; }

http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        proxy_connect_timeout 60;
        proxy_read_timeout 60;
        proxy_send_timeout 60;


        gzip on;
        gzip_disable "msie6";
        gzip_types text/plain text/xml text/css text/comma-separated-values;
        upstream app_server {
        server 127.0.0.1:8080 fail_timeout=0;
        }

#FAIL upstream app_server {server unix:/home/puma/sockets/puma.sock fail_timeout=0; }
# FAIL upstream app_server {server unix:/home/rails/tmp/sockets/unicorn.flossfish.sock fail_timeout=0; }
        include /etc/nginx/conf.d/*.conf;
       include /etc/nginx/sites-enabled/*;
}


nginx/sited-enabled/default
server {
        listen   80;
        root /home/rails/public;
        #server_name _;
        server_name localhost;
        index index.htm index.html;

        location / {
                try_files $uri/index.html $uri.html $uri @app;
        }

        location ~* ^.+\.(jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|mp3|flv|mpeg|avi)$ {
                        try_files $uri @app;
                }

         location @app {
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_redirect off;
                proxy_pass http://app_server;
    }

 error_page 500 502 503 504 /500.html;
    client_max_body_size 4G;
    keepalive_timeout 10;


}


unicorn.conf


listen "127.0.0.1:8080"
# FAIL listen "/home/rails/tmp/sockets/unicorn.flossfish.sock"

worker_processes 4
user "rails"
working_directory "/home/rails"
pid "/home/unicorn/pids/unicorn.pid"
stderr_path "/home/unicorn/logs/error.log"
stdout_path "/home/unicorn/logs/access.log"

timeout 30

----------------------------------------
netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      21572/unicorn.conf 
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      21546/nginx.conf


Grep nginx

ps -ef | grep nginx
root      9993  9242  0 Oct14 pts/2    00:00:00 tail -100f /var/log/nginx/error.log
root     21546     1  0 14:37 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
www-data 21549 21546  0 14:37 ?        00:00:00 nginx: worker process                   
www-data 21550 21546  0 14:37 ?        00:00:00 nginx: worker process                   
www-data 21551 21546  0 14:37 ?        00:00:00 nginx: worker process                   
www-data 21552 21546  0 14:37 ?        00:00:00 nginx: worker process


grep unicorn 
ps -ef | grep unicorn
root     20731  9242  0 13:52 pts/2    00:00:00 tail -f /home/unicorn/logs/error.log
root     21572     1  0 14:38 ?        00:00:00 unicorn master -D -c /home/unicorn/unicorn.conf -E production                                                                                               
rails    21575 21572  1 14:38 ?        00:00:04 unicorn worker[0] -D -c /home/unicorn/unicorn.conf -E production                                                                                            
rails    21578 21572  1 14:38 ?        00:00:04 unicorn worker[1] -D -c /home/unicorn/unicorn.conf -E production                                                                                            
rails    21580 21572  1 14:38 ?        00:00:04 unicorn worker[2] -D -c /home/unicorn/unicorn.conf -E production                                                                                            
rails    21583 21572  1 14:38 ?        00:00:04 unicorn worker[3] -D -c /home/unicorn/unicorn.conf -E production


 lsof -i :
nginx   16954     root    6u  IPv4 131527      0t0  TCP *:http (LISTEN)
nginx   16957 www-data    6u  IPv4 131527      0t0  TCP *:http (LISTEN)
nginx   16958 www-data    6u  IPv4 131527      0t0  TCP *:http (LISTEN)
nginx   16959 www-data    6u  IPv4 131527      0t0  TCP *:http (LISTEN)
nginx   16959 www-data    9u  IPv4 132064      0t0  TCP 104.236.247.33:http->118.101.25.229:54644 (ESTABLISHED)
nginx   16959 www-data   10u  IPv4 132065      0t0  TCP 104.236.247.33:http->118.101.25.229:54645 (ESTABLISHED)
nginx   16959 www-data   14u  IPv4 132074      0t0  TCP 104.236.247.33:http->112.208.71.167.pldt.net:62601(ESTABLISHED)
nginx   16960 www-data    6u  IPv4 131527      0t0  TCP *:http (LISTEN)
ruby    16984     root   10u  IPv4 131638      0t0  TCP localhost:http-alt (LISTEN)
ruby    16987    rails   10u  IPv4 131638      0t0  TCP localhost:http-alt (LISTEN)
ruby    16990    rails   10u  IPv4 131638      0t0  TCP localhost:http-alt (LISTEN)
ruby    16993    rails   10u  IPv4 131638      0t0  TCP localhost:http-alt (LISTEN)
ruby    16996    rails   10u  IPv4 131638      0t0  TCP localhost:http-alt (LISTEN)
3 Answers

Everything looks correct. The only thing I can think of is a firewall not letting nginx connect to unicorn. Can you post the output of the following command?

sudo iptables-save

Thanks Everyone. it turns out this was a rails/permission problem. I looked just about everywhere except in the unicorn access logs and thats where i discovered the problem. I think ultimately when the server was bounced I restarted the rails/unicorn processes and precompiled the rails assets it caused some folder permission issues at /tmp/cache/.... I set those permissions, chowed and viola everything stated working again! –

  • Thank you! My access logs also revealed the source of my problem causing refused connections: a rails bug due to a typo.

This solved my problem: https://viewsby.wordpress.com/2012/07/03/13permission-denied-proxy-http-attempt-to-connect-to-127-0-0-18080-localhost-failed/#comment-4233

I was trying to proxy Nginx https to Jenkins port 8080. I was able to remove the bad gateway error and the connection refused/permission denied error by running these commands on RedHat Linux:

/usr/sbin/setsebool httpdcannetwork_connect true
sudo service nginx restart

Have another answer? Share your knowledge.