Report this

What is the reason for this report?

Nginx + PHP FPM + MYSQL servers going down when too much traffic

Posted on November 24, 2017

We currently have some servers hosting a e-commerce platform in Digital Ocean, we tried to optimize how much we could, but was not enough, site is going down when too much traffic, like 800~1000 users. We currently think its the API that is going down. The error is:

2017/11/23 14:20:51 [error] 26784#26784: *126599 connect() to unix:/var/run/php/php7.0-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: {ip...}, server: www.....com.br, request: "GET /....", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "www......com.br"

We currently have:

Front-end server:

4 cores, 8gb ram - running nginx 1.10.3 and php fpm 7.0

API server:

8 cores, 16gb ram - running nginx 1.10.3 and php fpm 5.5.9

MYSQL server

MASTER + 8 SLAVES (We don’t think thats the problem), We had some problem before with connection numbers tcp/ip ports (I dont know exactly), but we could’t solve and created slaves for load balance. Servers are running linux ubuntu 14.04

If more information is needed, please ask. We are struggling with this.

Nginx conf:

worker_processes auto;
worker_rlimit_nofile 65536;
worker_connections 2048;
use epoll;
multi_accept on;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 15;
types_hash_max_size 2048;
server_names_hash_max_size 4112;
access_log off;
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 2;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;]
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
fastcgi_connect_timeout 360;
fastcgi_send_timeout 360;
fastcgi_read_timeout 360;
fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
client_body_buffer_size    128k;
client_max_body_size       10m;
client_header_buffer_size    1k;
large_client_header_buffers  4 4k;
output_buffers   1 32k;
postpone_output  1460;
client_header_timeout  3m;
client_body_timeout    3m;
send_timeout           3m;

PHP FPM conf:

pm = on demand;
pm.max_children = 200;
pm.process_idle_timeout = 10s;
pm.max_requests = 500;

/etc/sysctl.conf configs:

net.ipv4.ip_local_port_range = 2000 65535;
net.ipv4.tcp_rfc1337 = 1;
net.ipv4.tcp_fin_timeout = 15;
net.ipv4.tcp_keepalive_time = 300;
net.ipv4.tcp_keepalive_probes = 5;
net.ipv4.tcp_keepalive_intvl = 15;
net.core.rmem_default = 31457280;
net.core.rmem_max = 12582912;
net.core.wmem_default = 31457280;
net.core.wmem_max = 12582912;
net.core.somaxconn = 65535;
net.core.netdev_max_backlog = 65535;
net.core.optmem_max = 25165824;
net.ipv4.tcp_mem = 65535 131072 262144;
net.ipv4.udp_mem = 65535 131072 262144;
net.ipv4.tcp_rmem = 8192 87380 16777216;
net.ipv4.udp_rmem_min = 16384;
net.ipv4.tcp_wmem = 8192 65535 16777216;
net.ipv4.udp_wmem_min = 16384;
net.ipv4.tcp_max_tw_buckets = 1440000;
net.ipv4.tcp_tw_recycle = 1;
net.ipv4.tcp_tw_reuse = 1;


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Hello,

It sounds like that your PHP-FPM is crashing on your frontend server as the error that you’ve shared is referring to PHP 7. What is your current PHP max_memory limit?

I would suggest trying to add more RAM to your frontend server as this seems like your bottleneck at the comment.

Regards, Bobby

Heya,

Also an update on the topic. If you have such problems remember it’s always good to have SWAP. It won’t solve all your problems but it’s always a good idea to have one.

https://www.digitalocean.com/community/tutorial-collections/how-to-add-swap-space

Swap is a portion of hard drive storage that has been set aside for the operating system to temporarily store data that it can no longer hold in RAM. This lets you increase the amount of information that your server can keep in its working memory, with some caveats. The swap space on the hard drive will be used mainly when there is no longer sufficient space in RAM to hold in-use application data.

The information written to disk will be significantly slower than information kept in RAM, but the operating system will prefer to keep running application data in memory and use swap for the older data. Overall, having swap space as a fallback for when your system’s RAM is depleted can be a good safety net against out-of-memory exceptions on systems with non-SSD storage available.

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Get started for free

Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

*This promotional offer applies to new accounts only.