Related

Tool NGINX Config Generator Tool
DOCTL: DigitalOcean CLI Tool
Problem installing wordpress on subdomain (nginx / ubuntu) Question Question
Why are snapshots so slow Question Question

Question

connect() failed (111: Connection refused) while connecting to upstream

Posted August 14, 2019 83.8k views
NginxDigitalOceanAPILet's EncryptUbuntu 18.04

Hi all,

Followed this tutorial to get Let’s Encrypt working on my Nest.js/Node server.

Everything works - running sudo nginx -t returns:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

However, accessing my domain gives me a 502 Bad Gateway error. Specifically (from /var/log/nginx):

15004#15004: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 77.98.141.183, server: ww2.zone, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:8000/", host: "ww2.zone"

Here is my /etc/nginx/sites-available file:

Default server configuration
server {
    listen 80;
    listen 127.0.01;    
    listen [::]:80 default_server ipv6only=on;
    return 301 https://$host$request_uri;
}
server {
    # Enable HTTP/2
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name ww2.zone;

    # Use the Let’s Encrypt certificates
    ssl_certificate /etc/letsencrypt/live/ww2.zone/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ww2.zone/privkey.pem;

    # Include the SSL configuration from cipherli.st
    include snippets/ssl-params.conf;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-NginX-Proxy true;
        proxy_pass http://localhost:8000/;
        proxy_ssl_session_reuse off;
        proxy_set_header Host $http_host;
        proxy_cache_bypass $http_upgrade;
        proxy_redirect off;
    }
}

And my server is within a git Repo within home/username.

The code to listen on the port is:
await app.listen(process.env.PORT || 5000);

This used to be process.env.PORT || 80 but that would throw:

Error: listen EADDRINUSE: address already in use :::80

Any ideas? Pulling my hair out atm.

Add a comment
hbendixlewis
By:
hbendixlewis
Add a comment
1 comment
  • moisesalejandro September 23, 2020
    Show comment This comment has been marked as resolved by bobbyiliev.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
9 answers
bobbyiliev August 14, 2019

Hello,

You can only have 1 service listening on 1 port. So as you have Nginx listening on port 80 at the moment, you can not have your Node on the same port.

Also as you currently have your Nginx set to work as a reverse proxy for port 8000, what you could do is to just set Node to listen on port 8000.

This would fix your 502 error.

Hope that this helps!
Regards,
Bobby

Reply Report
  • hbendixlewis August 14, 2019

    Hi Bobby, appreciate the reply. I’ve just updated my Node server to listen on port 8000, however, I still get the same error!

    Reply Report
    • bobbyiliev August 14, 2019

      Hello,

      What I could suggest is checking the ports that are being used on your server with the netstat command. For example you could use the following:

      netstat -plant

      Make sure that Nginx is listening on 80 and 443 and that Node is listening on 8000 (as this is what you currently have in your Nginx config for your reverse proxy setup)

      Reply Report
      • hbendixlewis August 14, 2019

        Hi Bobby,

        Here’s the output table: 

        Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      -
tcp        0    816 167.71.143.98:22        77.98.141.183:50089     ESTABLISHED -
tcp        0      0 167.71.143.98:22        77.98.141.183:53826     ESTABLISHED -
tcp        0      0 167.71.143.98:22        77.98.141.183:53788     ESTABLISHED -
tcp        0      0 167.71.143.98:34772     104.248.175.171:25060   ESTABLISHED -
tcp        0   1080 167.71.143.98:22        153.36.236.35:41061     ESTABLISHED -
tcp6       0      0 :::80                   :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 :::443                  :::*                    LISTEN      -
tcp6       0      0 :::5000                 :::*                    LISTEN      -

        I’m struggling to determine what is the Nginx stuff vs what is my Node server? (I have the server running using Pm2.)

        Reply Report
        • bobbyiliev August 14, 2019

          Hello,

          I can see that the service is still listening on port 5000:

          tcp6       0      0 :::5000                 :::*                    LISTEN      -

          You need to either adjust that and set it to 8000, or you need to adjust your Nginx config and adjust the reverse proxy rule and change it from:

                  proxy_pass http://localhost:8000/;

          To:

                  proxy_pass http://localhost:5000/;

          That way Nginx would start proxying the connections from port 80 and port 443 to the service that is running on port 5000.

          Hope that this helps!

          Reply Report
  • loovum March 19, 2020

    I have the same problem as @hbendixlewis has.

    Here is my config. 

    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      960/mysqld
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      938/redis-server 12
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      21863/nginx: master
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      713/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      922/sshd
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      21863/nginx: master
tcp        0      0 127.0.0.1:3306          127.0.0.1:41634         TIME_WAIT   -
tcp        0    340 167.172.16.179:22       182.48.90.78:51251      ESTABLISHED 20492/sshd: root@pt
tcp6       0      0 ::1:6379                :::*                    LISTEN      938/redis-server 12
tcp6       0      0 :::80                   :::*                    LISTEN      21863/nginx: master
tcp6       0      0 :::22                   :::*                    LISTEN      922/sshd
tcp6       0      0 :::443                  :::*                    LISTEN      21863/nginx: master
    Reply Report
    • bobbyiliev March 26, 2020

      Hi there @loovum,

      Actually, I can’t see any services running on a specific port.

      Can you share more details of your exact setup? For example:

      • Share your Nginx Server Block config

      • What is the backend service that you are trying to connect to?

      • Make sure that your backend service is actually running

      Regards,
      Bobby

      Reply Report
  • chiuyau May 14, 2020

    Hi bobby, I have the same error but not sure if it is the same issue.
    I have the 502 page error on my site just after I installed php7.0 on my server. Before that, I am using Ghost theme and it’s application, I installed php7.0 for using php in the default page of nginx.
    Here’s the error log

    connect() failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xx.xx, server: example.com, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:2370/", host: "example.com"

    And the netstat -plant output table:

    Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:2368          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:2369          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:35602         127.0.0.1:3306          ESTABLISHED -
tcp        0      0 127.0.0.1:3306          127.0.0.1:35460         ESTABLISHED -
tcp        0      0 192.168.8.7:22          203.218.57.87:55401     ESTABLISHED -
tcp        0      0 127.0.0.1:35460         127.0.0.1:3306          ESTABLISHED -
tcp        0      0 127.0.0.1:35470         127.0.0.1:3306          ESTABLISHED -
tcp        0    376 192.168.8.7:22          203.218.57.87:55492     ESTABLISHED -
tcp        0      0 127.0.0.1:3306          127.0.0.1:35470         ESTABLISHED -
tcp        0      0 127.0.0.1:3306          127.0.0.1:35128         ESTABLISHED -
tcp        0      0 127.0.0.1:35128         127.0.0.1:3306          ESTABLISHED -
tcp        0      0 127.0.0.1:3306          127.0.0.1:35602         ESTABLISHED -
tcp6       0      0 :::443                  :::*                    LISTEN      -
tcp6       0      0 :::9000                 :::*                    LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      -
tcp6       0      0 :::21                   :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

    I am bad at troubleshooting, I am looking forward to your reply and I would be grateful. Thx!

    Reply Report
    • bobbyiliev May 16, 2020

      Hi there @chiuyau,

      It looks like that your backend service which is supposed to listen on port 2370 is not running.

      What are you using exactly for your backend services? It could be Docker or Node.js or something different, what you need to do is make sure that this service is up and running.

      I can see that you have some other services listening on ports 2368 and 2369 so my guess would be that the service which was listening on port 2370 had crashed or has been stopped for some other reasons and needs to be started again.

      Hope that this information gives you some ideas on where to start.

      Let me know how it goes!
      Regards,
      Bobby

      Reply Report
      • chiuyau May 16, 2020

        Thanks Bobby, I had solved the problem!
        For you reference, I’m using VMware and hosting the services local at my home, nodejs + ubuntu 18.04, though I’m not sure which service is listening 2368/2369, but I solve it by rewriting the nginx config to use ip, not dns. For instance, 127.0.0.1 instead of localhost, or remove the ipv6 alias from /etc/hosts.
        Thanks anyway!

        Reply Report
andersonfrfilho May 23, 2020

I have the same problem, when trying to access the nginx test page

2020/05/23 17:03:43 [error] 9037#9037: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 179.126.139.213, server: hellonode, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:3000/favicon.ico", host: "165.22.43.156", referrer: "http://165.22.43.156/"
2020/05/23 17:03:43 [error] 9037#9037: *27 connect() failed (111: Connection refused) while connecting to upstream, client: 179.126.139.213, server: hellonode, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "165.22.43.156"

run the comand netstat -plant

root@know-how-cloud:/# netstat -plant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 165.22.43.156:3333      0.0.0.0:*               LISTEN      6385/node /home/dep
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      9029/nginx: master
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      722/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      926/sshd
tcp        0      0 165.22.43.156:22        179.126.139.213:5270    ESTABLISHED 8011/sshd: root@not
tcp        0      0 165.22.43.156:37592     161.35.127.202:25060    ESTABLISHED 6385/node /home/dep
tcp        0    340 165.22.43.156:22        179.126.139.213:1979    ESTABLISHED 5495/sshd: deploy [
tcp        0   1081 165.22.43.156:22        112.85.42.195:41424     FIN_WAIT1   -
tcp        0      0 165.22.43.156:22        222.186.42.7:61212      ESTABLISHED 9455/sshd: [accepte
tcp        0      0 165.22.43.156:80        179.126.139.213:65126   ESTABLISHED 9037/nginx: worker
tcp        0      0 165.22.43.156:80        179.126.139.213:6467    ESTABLISHED 9037/nginx: worker
tcp6       0      0 :::80                   :::*                    LISTEN      9029/nginx: master
tcp6       0      0 :::22                   :::*                    LISTEN      926/sshd

Do you know how I can fix it?

Reply Report
  • bobbyiliev August 26, 2020

    Hi there,

    It looks like that you’ve specified port 3000 for your proxy pass. However, there is no service that is listening on that port.

    You either need to start your backend service on port 3000 or change the port in your Nginx config to match the port of your backend service.

    Regards,
    Bobby

    Reply Report
ihemantkumar June 10, 2020

I am having same issue however on browser, i am having this issue and logs doesn’t have such issues. I ran the command “netstat -plant”. Here’s the result.

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                   
tcp        0    612 <IP>:22        106.204.206.144:3884    ESTABLISHED -                   
tcp        0      1 <IP>:42432     <IP>:3306       SYN_SENT    -                   
tcp        0      1 <IP>:42434     <IP>:3306       SYN_SENT    -                   
tcp6       0      0 :::21                   :::*                    LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -
Reply Report
mukeshcoder1 August 26, 2020 
**Error:**  connect() failed (111: Connection refused) while connecting to upstream, 
client: 157.32.151.133, server: _, request: "GET /api/jinping/getdetails HTTP/1.1", 
upstream: "http://127.0.0.1:5000/api/jinping/getdetails", host: "104.248.238.219"


**NgInx Config File(/etc/nginx/sites-available): **

    upstream backend {
       server 127.0.0.1:5000
       keepalive 32;
    }


server {

server_name _;
listen 80;
location / {

proxy_pass http://127.0.0.1:5000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;

proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

}
#listen [::]:443; # managed by Certbot
#listen 443 ssl; # managed by Certbot
#ssl_certificate /etc/letsencrypt/live/your-domain/fullchain.pem; # managed by Certbot
#ssl_certificate_key /etc/letsencrypt/live/your-domain/privkey.pem; # managed by Certbot
#include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
#ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Service (.Net Core web API):

[Unit]
Description=HealthCare .NET Web API App running on Ubuntu

[Service]
WorkingDirectory=/var/www/healthcare/
ExecStart=/usr/bin/dotnet /var/www/healthcare/bin/Debug/netcoreapp2.0/publish/Session1.dll --urls="http://104.248.238.219:5000;https://104.248.238.219:5001"
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
SyslogIdentifier=Health
User=root
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false

[Install]
WantedBy=multi-user.target

Please help me to solve this issue.

edited by bobbyiliev
Reply Report
mukeshcoder1 August 26, 2020 
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN      19783/dotnet
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      26606/mysqld
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      14585/nginx: master
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      614/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      736/sshd
tcp        0      0 104.248.238.219:22      3.131.7.66:53227        ESTABLISHED 19114/sshd: root@no
tcp        0   1081 104.248.238.219:22      222.186.180.142:37107   FIN_WAIT1   -
tcp        0   1081 104.248.238.219:22      222.186.180.142:44469   FIN_WAIT1   -
tcp        0      1 104.248.238.219:22      222.186.169.192:47172   FIN_WAIT1   -
tcp        0      0 104.248.238.219:80      98.209.15.49:64734      SYN_RECV    -
tcp        0      0 104.248.238.219:22      113.57.170.50:18180     ESTABLISHED 19826/sshd: unknown
tcp        0      0 104.248.238.219:22      188.165.169.238:59054   ESTABLISHED 19829/sshd: unknown
tcp        0    464 104.248.238.219:22      3.131.7.66:53230        ESTABLISHED 19224/sshd: root@pt
tcp6       0      0 127.0.0.1:8005          :::*                    LISTEN      23842/java
tcp6       0      0 ::1:5000                :::*                    LISTEN      19783/dotnet
tcp6       0      0 :::8080                 :::*                    LISTEN      23842/java
tcp6       0      0 :::22                   :::*                    LISTEN      736/sshd

See this is my result after hit that command. I think service is starting there.

edited by bobbyiliev
Reply Report
amino123 August 26, 2020

hello, got same problem. But mine is very simple. My Webserver runs on my laptop, which is my editor for the www files. But i cannot visit my website from another Device in same network. it change the the local ip to localhost on my other device (?) or it gives me a 502 site

Since 4 hours i have been looking on the internet, did a lot of changes but nothing helped. Have you guys any tips for me ?

thanks and sorry for my bad english

Reply Report
amino123 August 29, 2020

have the same problem since a few days…

nginx conf

server {
  listen   80;
  root /var/www;
  index index.php index.html index.htm;
  server_name_in_redirect off;
  server_name localhost;
   proxy_redirect off;
  proxy_set_header Host $http_host; 

 location / {
      proxy_pass http://127.0.0.1:4001;
    proxy_set_header Host $server_name;
    try_files $uri $uri/ /index.php;
  }


#  error_page 404 /404.html;
#  error_page 500 502 503 504 /50x.html;
#  location = /50x.html {
#    root /var/www;
#  }

  # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
  location ~ \.php$ {
    try_files $uri =404;
    #fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
    fastcgi_pass 127.0.0.1:9000;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
  }
}

ufw

--                         ------      ---
80                         ALLOW       Anywhere                  
Nginx HTTP                 ALLOW       Anywhere                  
80 (v6)                    ALLOW       Anywhere (v6)             
Nginx HTTP (v6)            ALLOW       Anywhere (v6)

netstat -plant

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      783/systemd-resolve 
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      901/cupsd           
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      77280/php-fpm: mast 
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      68490/mysqld        
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      78190/nginx: master 

tcp6       0      0 ::1:631                 :::*                    LISTEN      901/cupsd           
tcp6       0      0 :::33060                :::*                    LISTEN      68490/mysqld

php www.conf

;listen = /var/run/php/php7.4-fpm.sock
listen = 127.0.0.1:9000

i dont know, even the people from stackoverflow (dont get me wrong)

hope you guys got the solution, thanks

edited by bobbyiliev
Reply Report
  • bobbyiliev August 31, 2020

    Hello,

    It looks like that you’ve setup a reverse proxy to proxy your traffic to port 4001 but at the moment nothing is running on that port.

    You need to make sure that your backend service is actually listening on port 4001.

    It would really depend on what exactly you are using for backend, for example, it might be a Docker container that has crashed on a Node.js service. But in any case you would need to get your backend service up and running.

    Let me know if you have any questions.
    Regards,
    Bobby

    Reply Report
amino123 August 31, 2020

hello bobbyiliev thank you for your reply.

i dont know what you mean with proxy 4001 and backend service, i have never heard of them. I got never issues like this, how i can i install and set up this kind of proxy? btw i run a shop system with admin interface.

these lines are only to stop nginx redirecting my other network devices to localhost

  proxy_pass http://127.0.0.1:4001;
    proxy_set_header Host $server_name;

hope you can help
regards
amino

Reply Report
  • bobbyiliev September 26, 2020

    Hi there @amino123,

    Yes so your current configuration is telling Nginx to proxy the traffic to http://127.0.0.1:4001, and in order for this to work, you need to have another service listening on that port.

    That service would really depend on your setup. For example, you might have a Node JS application running on that port or a Docker container and etc.

    From the netstat output that you’ve shared, I can see that there is no service listening on that port.

    In this case, you need to either remove the proxy_proxy pass rules completely or make sure that your service is started. But this would again really depend on your application and its architecture.

    Hope that this helps!
    Regards,
    Bobby

    Reply Report
bobbyiliev September 26, 2020

Hi there @moisesalejandro,

What I could suggest in your case is to check your Docker container’s logs for more information on why the connections might be failing.

To do that, you can run the following:

  • First get your Docker container ID:
  • docker ps -a
  • Then check the logs with:
  • docker logs your_container_id

Feel free to share the logs here!
Regards,
Bobby

Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help