nginx/1.4.6 (Ubuntu) server throwing 404 on all sites except just after reboot

July 28, 2015 19k views
Nginx

My server just started thorwing 404 on all sites (I'm running a few wordpress installs). htop shows no peaks in server load and when I reboot all sites work for about 2 minutes before throwing 404 again. I've made no changes to the server.

Anyone excperienced this? All help greatly appreciated.

6 comments
  • Is your php-fpm runnig? check service php-fpm status for it.

    Sounds like your php is not handling well all the requests. Also try looking into a caching solution like Varnish + Cloudflare

  • Hi @EpicCDN thanks for your reply, php5-fpm is running and I'm running cloudflare on all sites.

    I'm getting these errors in /var/log/nginx:

    2015/07/28 19:53:35 [error] 984#0: *858 connect() to unix:/var/run/php5-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 173.245.56.221, server: simmalugnt.se, request: "POST /wp-login.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "simmalugnt.se"
    2015/07/28 19:53:35 [error] 984#0: *858 open() "/usr/share/nginx/www/50x.html" failed (2: No such file or directory), client: 173.245.56.221, server: simmalugnt.se, request: "POST /wp-login.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "simmalugnt.se"
    

    and

    2015/07/28 20:05:01 [crit] 9630#0: *101 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 141.101.99.233, server: simmalugnt.se, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "simmalugnt.se", referrer: "http://simmalugnt.se/"
    
  • Yeah, your php-fpm is failing so you need to find out why, ie is the php running out of memory? or there is some other issue, probably the first one.

    can you copy/paste your nginx (one of the sites or two of them should be better) and your php-fpm www.conf?

    Also do you have a single pool (php-fpm) for all your sites or use multiple pools one for each site?

  • Sure, here is my /etc/php5/fpm/pool.d/www.conf:

    ; Start a new pool named 'www'.
    ; the variable $pool can we used in any directive and will be replaced by the
    ; pool name ('www' here)
    [www]
    
    ; Unix user/group of processes
    ; Note: The user is mandatory. If the group is not set, the default user's group
    ;       will be used.
    user = www-data
    group = www-data
    
    ; The address on which to accept FastCGI requests.
    ; Valid syntaxes are:
    ;   'ip.add.re.ss:port'    - to listen on a TCP socket to a specific address on
    ;                            a specific port;
    ;   'port'                 - to listen on a TCP socket to all addresses on a
    ;                            specific port;
    ;   '/path/to/unix/socket' - to listen on a unix socket.
    ; Note: This value is mandatory.
    listen = /var/run/php5-fpm.sock
    
    ; Set listen(2) backlog.
    ; Default Value: 65535 (-1 on FreeBSD and OpenBSD)
    ;listen.backlog = 65535
    
    ; Set permissions for unix socket, if one is used. In Linux, read/write
    ; permissions must be set in order to allow connections from a web server. Many
    ; BSD-derived systems allow connections regardless of permissions.
    ; Default Values: user and group are set as the running user
    ;                 mode is set to 0660
    listen.owner = www-data
    listen.group = www-data
    listen.mode = 0660
    [...]
    

    Here is a site:

    server {
        # URL: Correct way to redirect URL's
        server_name www.simmalugnt.se;
        rewrite ^/(.*)$ http://simmalugnt.se/$1 permanent;
    }
    server {
        server_name simmalugnt.se new.simmalugnt.se;
        root /var/www/simmalugnt;
        access_log /var/log/nginx/www.simmalugnt.se.access.log;
        error_log /var/log/nginx/www.simmalugnt.se.error.log;
        include global/common.conf;
        include global/wordpress.conf;
    }
    

    Here are the included configfiles:

    # Global configuration file.
    # ESSENTIAL : Configure Nginx Listening Port
    listen 80;
    # ESSENTIAL : Default file to serve. If the first file isn't found,
    index index.php index.html index.htm;
    # ESSENTIAL : no favicon logs
    location = /favicon.ico {
        log_not_found off;
        access_log off;
    }
    # ESSENTIAL : robots.txt
    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }
    # ESSENTIAL : Configure 404 Pages
    error_page 404 /404.html;
    # ESSENTIAL : Configure 50x Pages
    error_page 500 502 503 504 /50x.html;
        location = /50x.html {
            root /usr/share/nginx/www;
        }
    # SECURITY : Deny all attempts to access hidden files .abcde
    location ~ /\. {
        deny all;
    }
    # PERFORMANCE : Set expires headers for static files and turn off logging.
    location ~* ^.+\.(js|css|swf|xml|txt|ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|rss|atom|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$ {
        access_log off; log_not_found off; expires 30d;
    }
    
    # WORDPRESS : Rewrite rules, sends everything through index.php and keeps the appended query string intact
    location / {
        try_files $uri $uri/ /index.php?q=$uri&$args;
    }
    
    # SECURITY : Deny all attempts to access PHP Files in the uploads directory
    location ~* /(?:uploads|files)/.*\.php$ {
        deny all;
    }
    # REQUIREMENTS : Enable PHP Support
    location ~ \.php$ {
        # SECURITY : Zero day Exploit Protection
        try_files $uri =404;
        # ENABLE : Enable PHP, listen fpm sock
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
    }
    # PLUGINS : Enable Rewrite Rules for Yoast SEO SiteMap
    rewrite ^/sitemap_index\.xml$ /index.php?sitemap=1 last;
    rewrite ^/([^/]+?)-sitemap([0-9]+)?\.xml$ /index.php?sitemap=$1&sitemap_n=$2 last;
    
    #Yeah! you did it.
    

    I'm a bit of a novice at this but I think I'm using just one pool for all sites, but htop shows no big server load..

  • Sorry I just read your reply, but www.conf is incomplete, please paste that complete as is the most important part here.

  • @EpicCDN cheers but the issue had been resolved now (see solution below), thank you so much for your help!

4 Answers

@jens834585 we need to change your System. Here you go :

Before you start Snapshoot your Droplet

Paste this (at the end of the file) to /etc/sysctl.conf (using nano /etc/sysctl.conf) and save it (this change not only fixes your error it makes your System much better),

### IMPROVE SYSTEM MEMORY MANAGEMENT ###

# Increase size of file handles and inode cache
fs.file-max = 2097152

# Do less swapping
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2

### GENERAL NETWORK SECURITY OPTIONS ###

# Number of times SYNACKs for passive TCP connection.
net.ipv4.tcp_synack_retries = 2

# Allowed local port range
net.ipv4.ip_local_port_range = 2000 65535

# Protect Against TCP Time-Wait
net.ipv4.tcp_rfc1337 = 1

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for connections to keep alive
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

### TUNING NETWORK PERFORMANCE ###

# Default Socket Receive Buffer
net.core.rmem_default = 31457280

# Maximum Socket Receive Buffer
net.core.rmem_max = 12582912

# Default Socket Send Buffer
net.core.wmem_default = 31457280

# Maximum Socket Send Buffer
net.core.wmem_max = 12582912

# Increase number of incoming connections
net.core.somaxconn = 65535

# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 65535

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 25165824

# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 65535 131072 262144
net.ipv4.udp_mem = 65535 131072 262144

# Increase the read-buffer space allocatable
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.udp_rmem_min = 16384

# Increase the write-buffer-space allocatable
net.ipv4.tcp_wmem = 8192 65535 16777216
net.ipv4.udp_wmem_min = 16384

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1

Paste this (at the end of the file) to /etc/security/limits.conf (using nano /etc/security/limits.conf) and save it,

nginx           soft    nofile          2097152
nginx           hard    nofile          2097152
root            soft    nofile          2097152
root            hard    nofile          2097152
www-data        soft    nofile          2097152
www-data        hard    nofile          2097152

Paste this (at the end of the file) to /etc/pam.d/common-session (using nano /etc/pam.d/common-session) and save it,

session required pam_limits.so

Change listen.backlog in /etc/php5/fpm/pool.d/www.conf (using nano /etc/php5/fpm/pool.d/www.conf) and save it,

listen.backlog = 65535

Change workerrlimitnofile in /etc/nginx/nginx.conf (using nano /etc/nginx/nginx.conf) and save it,

worker_rlimit_nofile 99999;

And finally reboot

PS: If it works give a hearth to show me some love

  • Hi @eldin - thank you so much for your response. I've done the changes you suggested and it seems to have made some difference, however the sites still stop working after a while (5 minutes) and before that they seem to get slower and slower (until they stop working alltogether).

    I no longer get this error in the log:

    2015/07/28 19:53:35 [error] 984#0: *858 connect() to unix:/var/run/php5-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 173.245.56.221, server: simmalugnt.se, request: "POST /wp-login.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "simmalugnt.se"
    

    Only this:

    2015/07/29 10:53:47 [error] 1855#0: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 141.101.106.155, server: simmalugnt.se, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "simmalugnt.se", referrer: "http://simmalugnt.se/"
    

    EDIT
    After another reboot the sites have been running fine for 30 minutes, maybe the problem is resolved after all. I'll update again in a while.

    EDIT 2
    The server has been running now for almost an hour and all sites are still working fine. Thank you @eldin - you're a lifesaver!

  • Hi, @eldin

    Are your programs suitable for other types of VPS?

    If the RAM is 512MB or RAM 1G of VPS, suitable for those changes?
     
    Is your method can enhance their performance?

    Thanks!!

    • Hello @IIPPs ,

      you can do this on any VPS or Bare metal Server :-)

      This does not make your Server faster but smarter. Hey that is a rhyme :-)

      PS: This is not a program it's a tweak! The main part for NGINX/PHP is :

      net.core.somaxconn = 65535
      

Hi

I am facing same issue . tried all given in the forum yet it is showing 404 :'

Have another answer? Share your knowledge.