I have a droplet with nginx/node, some requests don't get to the server? Any ideas how to debug?

Question

I have a nodejs server behind nginx.
Sometimes, http requests get “timeout”.
Checking both the access and error logs for both the nginx and node does’t even show the requests coming to either of them, let alone fail.
Some of these requests are from the server itself (using curl via crontab).

I’m lost on how to even begin to start the problem. Any ideas?

Answer 1

gfelot33 December 6, 2016

Can you post your /etc/nginx/site-enabled/ files ?

Reply Report

RaniSharim December 6, 2016
upstream my.domain.here {
server 127.0.0.1:3456;
}

server {
listen 0.0.0.0:80;
servername my.domain.here;
accesslog /var/log/nginx/my.domain.here.access.log;
error_log /var/log/nginx/my.domain.here.error.log debug;

location / {
proxysetheader X-Real-IP $remoteaddr;
proxysetheader X-Forwarder-For $proxyaddxforwardedfor;
proxysetheader Host $httphost;
proxysetheader X-NginX-Proxy true;

proxy_pass http://my.domain.here; proxy_redirect off; proxy_read_timeout 600;

}
}

server {
listen 0.0.0.0:443;
servername my.domain.here;
accesslog /var/log/nginx/ssl.my.domain.here.access.log;
error_log /var/log/nginx/ssl.my.domain.here.error.log debug;

ssl on; ssl_certificate /etc/letsencrypt/live/my.domain.here/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/my.domain.here/privkey.pem; ssl_trusted_certificate /etc/letsencrypt/live/my.domain.here/chain.pem; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_prefer_server_ciphers on; ssl_session_timeout 1d; ssl_session_cache shared:SSL:50m; ssl_stapling on; ssl_stapling_verify on; add_header Strict-Transport-Security max-age=15768000;

location / {
proxysetheader X-Real-IP $remoteaddr;
proxysetheader X-Forwarder-For $proxyaddxforwardedfor;
proxysetheader Host $httphost;
proxysetheader X-NginX-Proxy true;

proxy_pass http://my.domain.here; proxy_redirect off; proxy_read_timeout 600;

}

}
Reply Report
- gfelot33 December 7, 2016
  
  Your nginx blocks looks good to me. Even if I would advice you to use your https connection all the time by redirecting (301) every http request to https.
  
  On my own server I start my block like this :
  
  server { listen 80; listen [::]:80 #for ipV6 if available server_name www.yourDN.com yourDN.com; return 301 https://yourDN.com$request_uri; } server { listen 443 ssl http2 default_server; listen [::]:443 ssl http2 default_server server_name yourDN.com; [...] }
  
  Reply Report

Answer 2

RaniSharim December 6, 2016
upstream my.domain.here {
server 127.0.0.1:3456;
}

server {
listen 0.0.0.0:80;
servername my.domain.here;
accesslog /var/log/nginx/my.domain.here.access.log;
error_log /var/log/nginx/my.domain.here.error.log debug;

location / {
proxysetheader X-Real-IP $remoteaddr;
proxysetheader X-Forwarder-For $proxyaddxforwardedfor;
proxysetheader Host $httphost;
proxysetheader X-NginX-Proxy true;

proxy_pass http://my.domain.here; proxy_redirect off; proxy_read_timeout 600;

}
}

server {
listen 0.0.0.0:443;
servername my.domain.here;
accesslog /var/log/nginx/ssl.my.domain.here.access.log;
error_log /var/log/nginx/ssl.my.domain.here.error.log debug;

ssl on; ssl_certificate /etc/letsencrypt/live/my.domain.here/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/my.domain.here/privkey.pem; ssl_trusted_certificate /etc/letsencrypt/live/my.domain.here/chain.pem; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_prefer_server_ciphers on; ssl_session_timeout 1d; ssl_session_cache shared:SSL:50m; ssl_stapling on; ssl_stapling_verify on; add_header Strict-Transport-Security max-age=15768000;

location / {
proxysetheader X-Real-IP $remoteaddr;
proxysetheader X-Forwarder-For $proxyaddxforwardedfor;
proxysetheader Host $httphost;
proxysetheader X-NginX-Proxy true;

proxy_pass http://my.domain.here; proxy_redirect off; proxy_read_timeout 600;

}

}
Reply Report
- gfelot33 December 7, 2016
  
  Your nginx blocks looks good to me. Even if I would advice you to use your https connection all the time by redirecting (301) every http request to https.
  
  On my own server I start my block like this :
  
  server { listen 80; listen [::]:80 #for ipV6 if available server_name www.yourDN.com yourDN.com; return 301 https://yourDN.com$request_uri; } server { listen 443 ssl http2 default_server; listen [::]:443 ssl http2 default_server server_name yourDN.com; [...] }
  
  Reply Report

Answer 3

xMudrii December 6, 2016

I would make sure that your connection from which you test is stable enough. Unstable connection can cause that request you sent doesn’t even get send. You can create script that will curl server and run it from multiple workstations/connections. Ping could also be good to test server responsiveness but curl would be better if available.

From server side you can verify that Droplet isn’t under heavy load. Check CPU and RAM. Test above and this in same time, you can get some pattern that’ll help debug issue. You can also see current I/O Bandwidth from DO Control Panel.

Reply Report

RaniSharim December 6, 2016

Some of the lost requests are from the server to itself, so unstable connection is unlikely.

There isn’t heavy load on the server, and the requests were lost in off-peak hours.
But I’ll check i/o and cpu in the dashboard, thanks. (Edit: checked, very light load in these hours)

Also, wouldn’t nginx report failures due to heavy load?
Reply Report
- xMudrii December 6, 2016
  
  I understood that you have problem from anywhere, my bad.
  
  Server load is not the problem. As CPU is not heavy loaded, it doesn’t make any problem. If RAM was problem, it’s very likely that nginx would got killed, so we can remove RAM from problem list.
  
  What you can try next? You can try writing curl verbose output to some log file.
  
  If you execute curl --verbose ... you can get more details. Because you’re using corn job, output is not visible in terminal, we need to write it somewhere.
  
  If you have cron job like this * * * * * curl ..., add verbose flag and log location:
  * * * * * curl --verbose ... >> /var/log/curl-nginx.log.
  
  You can set any location for log, just make sure you have write permissions there.
  >> means redirect output and append it to file.
  --verbose or also -v should give you bit more details on request.
  
  Reply Report

Answer 4

RaniSharim December 6, 2016

Some of the lost requests are from the server to itself, so unstable connection is unlikely.

There isn’t heavy load on the server, and the requests were lost in off-peak hours.
But I’ll check i/o and cpu in the dashboard, thanks. (Edit: checked, very light load in these hours)

Also, wouldn’t nginx report failures due to heavy load?
Reply Report
- xMudrii December 6, 2016
  
  I understood that you have problem from anywhere, my bad.
  
  Server load is not the problem. As CPU is not heavy loaded, it doesn’t make any problem. If RAM was problem, it’s very likely that nginx would got killed, so we can remove RAM from problem list.
  
  What you can try next? You can try writing curl verbose output to some log file.
  
  If you execute curl --verbose ... you can get more details. Because you’re using corn job, output is not visible in terminal, we need to write it somewhere.
  
  If you have cron job like this * * * * * curl ..., add verbose flag and log location:
  * * * * * curl --verbose ... >> /var/log/curl-nginx.log.
  
  You can set any location for log, just make sure you have write permissions there.
  >> means redirect output and append it to file.
  --verbose or also -v should give you bit more details on request.
  
  Reply Report

Related

Question

I have a droplet with nginx/node, some requests don't get to the server? Any ideas how to debug?

Related

Leave a comment

Looking for something else?

Sign up for our newsletter

Related

Question

I have a droplet with nginx/node, some requests don't get to the server? Any ideas how to debug?

Related

Leave a comment

Related

question

Is both HTTP and HTTPS versions should be verified in Webmaster Tools?

question

subdomain pointing to Digital Ocean and need to send email

question

my site i down please see http://www.cheri3a.com/

question

How do I get a SSL certificate for my WSGI Flask server so it can send data over HTTPS?

Looking for something else?