Having problem with Php fpm

Question

Ok My site is working fine but when the number of visitors increases it breaks suddenly.To find out why it breaks I have been observing the behaviour of my server using the ‘htop’ command.

Using 'htop’ I noticed that whenever the second value in the figure,

code Tasks 136 , 100 , 2 running

reaches 166 my site breaks,I get something like “ 502 Gateway error” .The problem seems to be not coming from RAM because the RAM usage is just 4 to 5 GB but I have 16GB RAM.And also the CPU usage never reaches 100%.So where is the problem coming from? can anyone help me please!

My Server is information:

16GB RAM, 320 GB storage , 6 CPU

My NGinx configuration is :

worker_processes 6;
worker_connections 2000;

My PHP fpm configuration is :

pm = ondemand
pm.max_children = 100
pm.start_servers = 18
pm.min_spare_servers = 9
pm.max_spare_servers = 18
pm.process_idle_timeout = 10s;
pm.max_requests = 500

Am I missing something,I also use WP Super cache!!

Any help will be grateful,thank you in advance.

Answer 1

unixynet October 2, 2018

Increase max_children to 150 and restart fpm. Also, check the php-fpm error log. It should have further clues on why it’s failing.

Reply Report

NewCustomer October 3, 2018
I have increased the max_children to 150 but the problem still exists.Now instead of 166 my site breaks at 216.This does not mean my site can support more visitors than before.

The max visitors supported when the maxchildren value was 100 is same as the max visitors supported when the maxchildren is set to 150. So no improvement in the performance of the site.

Now the only thing that change are the first and second value in the figure :

code Tasks 136 , 100 , 2 running

to max value of 156 and 216 .Note there is still no improvement in the performance it is same as before.
Reply Report
- unixynet October 3, 2018
  
  What’s in the php fpmlog when the 502 error happens?
  
  Reply Report
  
  NewCustomer October 3, 2018
  
  Here is the log:
  
  [03-Oct-2018 13:09:34] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 10 idle, and 138 total children [03-Oct-2018 13:09:35] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 10 idle, and 139 total children [03-Oct-2018 13:09:36] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 6 idle, and 140 total children [03-Oct-2018 13:09:46] WARNING: [pool www] server reached pm.max_children setting (150), consider raising it [03-Oct-2018 13:17:31] WARNING: [pool www] server reached pm.max_children setting (150), consider raising it
  
  Reply Report
  
  unixynet October 3, 2018
  
  So you have two things to address
  
  You need to increase max_children again. Try 230
  
  Set start_servers to 50
  
  Try to see how much memory your php processes are consuming per run on average. So if it’s 60MB on average, that tells you can increase maxchildren to a theoretical maximum of 16000/60 = 266. Your server will of course need to reserve RAM for other services. So perhaps 250 is your maximum maxchildren number.
  
  Reply Report
  
  NewCustomer October 5, 2018
  
  No changes still the same behaviour!!
  
  Reply Report
  
  unixynet October 5, 2018
  
  Check the logs again to see if there are any warnings like the ones above
  
  Reply Report
  
  NewCustomer October 7, 2018
  
  Now I am getting “Error establishing database connections” on reaching the maximum value of 215 and 216 for the first and second value in the figure:
  
  Tasks 215 , 216 , 2 running
  
  Reply Report
  
  unixynet October 7, 2018
  
  Make sure max_connections is set to 250 in your my.cnf. Then restart MySQL.
  
  Unlimited Servers. Unlimited Requests. https://www.unixy.net/server-management
  
  Reply Report
  
  NewCustomer October 10, 2018
  
  NO man it’s still not working!!
  
  One thing I found out that whenever the tasks reaches max, in the front-end I get “Error establishing database connection” any idea why it is happening!!
  
  Reply Report

Answer 2

NewCustomer October 3, 2018
I have increased the max_children to 150 but the problem still exists.Now instead of 166 my site breaks at 216.This does not mean my site can support more visitors than before.

The max visitors supported when the maxchildren value was 100 is same as the max visitors supported when the maxchildren is set to 150. So no improvement in the performance of the site.

Now the only thing that change are the first and second value in the figure :

code Tasks 136 , 100 , 2 running

to max value of 156 and 216 .Note there is still no improvement in the performance it is same as before.
Reply Report
- unixynet October 3, 2018
  
  What’s in the php fpmlog when the 502 error happens?
  
  Reply Report
  
  NewCustomer October 3, 2018
  
  Here is the log:
  
  [03-Oct-2018 13:09:34] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 10 idle, and 138 total children [03-Oct-2018 13:09:35] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 10 idle, and 139 total children [03-Oct-2018 13:09:36] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 6 idle, and 140 total children [03-Oct-2018 13:09:46] WARNING: [pool www] server reached pm.max_children setting (150), consider raising it [03-Oct-2018 13:17:31] WARNING: [pool www] server reached pm.max_children setting (150), consider raising it
  
  Reply Report
  
  unixynet October 3, 2018
  
  So you have two things to address
  
  You need to increase max_children again. Try 230
  
  Set start_servers to 50
  
  Try to see how much memory your php processes are consuming per run on average. So if it’s 60MB on average, that tells you can increase maxchildren to a theoretical maximum of 16000/60 = 266. Your server will of course need to reserve RAM for other services. So perhaps 250 is your maximum maxchildren number.
  
  Reply Report
  
  NewCustomer October 5, 2018
  
  No changes still the same behaviour!!
  
  Reply Report
  
  unixynet October 5, 2018
  
  Check the logs again to see if there are any warnings like the ones above
  
  Reply Report
  
  NewCustomer October 7, 2018
  
  Now I am getting “Error establishing database connections” on reaching the maximum value of 215 and 216 for the first and second value in the figure:
  
  Tasks 215 , 216 , 2 running
  
  Reply Report
  
  unixynet October 7, 2018
  
  Make sure max_connections is set to 250 in your my.cnf. Then restart MySQL.
  
  Unlimited Servers. Unlimited Requests. https://www.unixy.net/server-management
  
  Reply Report
  
  NewCustomer October 10, 2018
  
  NO man it’s still not working!!
  
  One thing I found out that whenever the tasks reaches max, in the front-end I get “Error establishing database connection” any idea why it is happening!!
  
  Reply Report

Answer 3

unixynet October 10, 2018

It’s happening because your max_connections is being reached. Run these commands and copy/paste the output here:

wget http://mysqltuner.pl
mv index.html mysqltuner.pl
perl mysqltuner.pl

On the last command, you might need to run it like this:

perl mysqltuner.pl --user root -pass <mysql_root_passwd>

Reply Report

NewCustomer October 10, 2018

——– Log file Recommendations ——————————————————————
[–] Log file: /var/log/mysql/error.log(106K)
[OK] Log file /var/log/mysql/error.log exists
[OK] Log file /var/log/mysql/error.log is readable.
[OK] Log file /var/log/mysql/error.log is not empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[OK] /var/log/mysql/error.log doesn’t contain any warning.
[!!] /var/log/mysql/error.log contains 291 error(s).
[–] 0 start(s) detected in /var/log/mysql/error.log
[–] 0 shutdown(s) detected in /var/log/mysql/error.log

——– Storage Engine Statistics —————————————————————–
[–] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRGMYISAM +MyISAM +PERFORMANCESCHEMA
[–] Data in InnoDB tables: 90.9M (Tables: 41)
[OK] Total fragmented tables: 0

——– Analysis Performance Metrics ————————————————————–
[–] innodbstatsonmetadata: OFF
[OK] No stat updates during querying INFORMATIONSCHEMA.

——– Security Recommendations ——————————————————————
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!

——– CVE Security Recommendations ————————————————————–
[–] Skipped due to –cvefile option undefined

——– Performance Metrics ———————————————————————–
[–] Up for: 15d 5h 52m 5s (413M q [313.895 qps], 10M conn, TX: 13927G, RX: 46G)
[–] Reads / Writes: 99% / 1%
[–] Binary logging is disabled
[–] Physical Memory : 15.7G
[–] Max MySQL memory : 352.4M
[–] Other process memory: 12.5G
[–] Total buffers: 192.0M global + 1.1M per thread (151 max threads)
[–] P_S Max memory usage: 72B
[–] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 353.5M (2.20% of installed RAM)
[OK] Maximum possible memory usage: 352.4M (2.20% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (0/413M)
[!!] Highest connection usage: 100% (152/151)
[!!] Aborted connections: 3.24% (330353/10202656)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 0.0% (0 cached / 362M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 1% (799K temp sorts / 51M sorts)
[OK] No joins without indexes
[!!] Temporary tables created on disk: 71% (14M on disk / 20M total)
[OK] Thread cache hit rate: 97% (228K created / 10M connections)
[OK] Table cache hit rate: 22% (398 open / 1K opened)
[OK] Open file limit used: 0% (0/1K)
[OK] Table locks acquired immediately: 100% (2K immediate / 2K locks)

——– Performance schema ————————————————————————
[–] Memory used by P_S: 72B
[–] Sys schema is installed.

——– ThreadPool Metrics ————————————————————————
[–] ThreadPool stat is disabled.

——– MyISAM Metrics —————————————————————————-
[!!] Key buffer used: 18.2% (3M used / 16M cache)
[OK] Key buffer size / total MyISAM indexes: 16.0M/43.0K
[!!] Read Key buffer hit rate: 85.2% (88 cached / 13 reads)
[OK] Write Key buffer hit rate: 100.0% (10 cached / 10 writes)

——– InnoDB Metrics —————————————————————————-
[–] InnoDB is enabled.
[–] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 128.0M/90.9M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal 25%
[OK] InnoDB buffer pool instances: 1
[–] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodbbufferpoolsize aligned with Innodbbufferpoolchunksize & Innodbbufferpoolinstances
[OK] InnoDB Read buffer efficiency: 100.00% (48248427167 hits/ 48248428524 total)
[!!] InnoDB Write Log efficiency: 10.35% (122327 hits/ 1181608 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1059281 writes)

——– AriaDB Metrics —————————————————————————-
[–] AriaDB is disabled.

——– TokuDB Metrics —————————————————————————-
[–] TokuDB is disabled.

——– XtraDB Metrics —————————————————————————-
[–] XtraDB is disabled.

——– Galera Metrics —————————————————————————-
[–] Galera is disabled.

——– Replication Metrics ———————————————————————–
[–] Galera Synchronous replication: NO
[–] No replication slave(s) for this server.
[–] Binlog format: ROW
[–] XA support enabled: ON
[–] Semi synchronous replication Master: Not Activated
[–] Semi synchronous replication Slave: Not Activated
[–] This is a standalone server

——– Recommendations —————————————————————————
General recommendations:
Control error line(s) into /var/log/mysql/error.log file
Reduce or eliminate persistent connections to reduce connection usage
Reduce or eliminate unclosed connections and network issues
Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
When making adjustments, make tmptablesize/maxheaptablesize equal
Reduce your SELECT DISTINCT queries which have no LIMIT clause
Before changing innodblogfilesize and/or innodblogfilesingroup read this: http://bit.ly/2wgkDvS
Variables to adjust:
maxconnections (> 151)
waittimeout (< 28800)
interactivetimeout (< 28800)
querycachesize (=0)
querycachetype (=0)
querycachelimit (> 1M, or use smaller result sets)
tmptablesize (> 16M)
maxheaptablesize (> 16M)
innodblogfile_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
Reply Report
- unixynet October 10, 2018
  
  I thought you changed max_connections to 200+ in my.cnf:
  
  [!!] Highest connection usage: 100% (152/151) [!!] Aborted connections: 3.24% (330353/10202656)
  
  Right now this is the culprit that’s causing your DB error. Make sure you change it and then restart mysql.
  
  Reply Report

Answer 4

NewCustomer October 10, 2018

——– Log file Recommendations ——————————————————————
[–] Log file: /var/log/mysql/error.log(106K)
[OK] Log file /var/log/mysql/error.log exists
[OK] Log file /var/log/mysql/error.log is readable.
[OK] Log file /var/log/mysql/error.log is not empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[OK] /var/log/mysql/error.log doesn’t contain any warning.
[!!] /var/log/mysql/error.log contains 291 error(s).
[–] 0 start(s) detected in /var/log/mysql/error.log
[–] 0 shutdown(s) detected in /var/log/mysql/error.log

——– Storage Engine Statistics —————————————————————–
[–] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRGMYISAM +MyISAM +PERFORMANCESCHEMA
[–] Data in InnoDB tables: 90.9M (Tables: 41)
[OK] Total fragmented tables: 0

——– Analysis Performance Metrics ————————————————————–
[–] innodbstatsonmetadata: OFF
[OK] No stat updates during querying INFORMATIONSCHEMA.

——– Security Recommendations ——————————————————————
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!

——– CVE Security Recommendations ————————————————————–
[–] Skipped due to –cvefile option undefined

——– Performance Metrics ———————————————————————–
[–] Up for: 15d 5h 52m 5s (413M q [313.895 qps], 10M conn, TX: 13927G, RX: 46G)
[–] Reads / Writes: 99% / 1%
[–] Binary logging is disabled
[–] Physical Memory : 15.7G
[–] Max MySQL memory : 352.4M
[–] Other process memory: 12.5G
[–] Total buffers: 192.0M global + 1.1M per thread (151 max threads)
[–] P_S Max memory usage: 72B
[–] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 353.5M (2.20% of installed RAM)
[OK] Maximum possible memory usage: 352.4M (2.20% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (0/413M)
[!!] Highest connection usage: 100% (152/151)
[!!] Aborted connections: 3.24% (330353/10202656)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 0.0% (0 cached / 362M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 1% (799K temp sorts / 51M sorts)
[OK] No joins without indexes
[!!] Temporary tables created on disk: 71% (14M on disk / 20M total)
[OK] Thread cache hit rate: 97% (228K created / 10M connections)
[OK] Table cache hit rate: 22% (398 open / 1K opened)
[OK] Open file limit used: 0% (0/1K)
[OK] Table locks acquired immediately: 100% (2K immediate / 2K locks)

——– Performance schema ————————————————————————
[–] Memory used by P_S: 72B
[–] Sys schema is installed.

——– ThreadPool Metrics ————————————————————————
[–] ThreadPool stat is disabled.

——– MyISAM Metrics —————————————————————————-
[!!] Key buffer used: 18.2% (3M used / 16M cache)
[OK] Key buffer size / total MyISAM indexes: 16.0M/43.0K
[!!] Read Key buffer hit rate: 85.2% (88 cached / 13 reads)
[OK] Write Key buffer hit rate: 100.0% (10 cached / 10 writes)

——– InnoDB Metrics —————————————————————————-
[–] InnoDB is enabled.
[–] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 128.0M/90.9M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal 25%
[OK] InnoDB buffer pool instances: 1
[–] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodbbufferpoolsize aligned with Innodbbufferpoolchunksize & Innodbbufferpoolinstances
[OK] InnoDB Read buffer efficiency: 100.00% (48248427167 hits/ 48248428524 total)
[!!] InnoDB Write Log efficiency: 10.35% (122327 hits/ 1181608 total)
[OK] InnoDB log waits: 0.00% (0 waits / 1059281 writes)

——– AriaDB Metrics —————————————————————————-
[–] AriaDB is disabled.

——– TokuDB Metrics —————————————————————————-
[–] TokuDB is disabled.

——– XtraDB Metrics —————————————————————————-
[–] XtraDB is disabled.

——– Galera Metrics —————————————————————————-
[–] Galera is disabled.

——– Replication Metrics ———————————————————————–
[–] Galera Synchronous replication: NO
[–] No replication slave(s) for this server.
[–] Binlog format: ROW
[–] XA support enabled: ON
[–] Semi synchronous replication Master: Not Activated
[–] Semi synchronous replication Slave: Not Activated
[–] This is a standalone server

——– Recommendations —————————————————————————
General recommendations:
Control error line(s) into /var/log/mysql/error.log file
Reduce or eliminate persistent connections to reduce connection usage
Reduce or eliminate unclosed connections and network issues
Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
When making adjustments, make tmptablesize/maxheaptablesize equal
Reduce your SELECT DISTINCT queries which have no LIMIT clause
Before changing innodblogfilesize and/or innodblogfilesingroup read this: http://bit.ly/2wgkDvS
Variables to adjust:
maxconnections (> 151)
waittimeout (< 28800)
interactivetimeout (< 28800)
querycachesize (=0)
querycachetype (=0)
querycachelimit (> 1M, or use smaller result sets)
tmptablesize (> 16M)
maxheaptablesize (> 16M)
innodblogfile_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
Reply Report
- unixynet October 10, 2018
  
  I thought you changed max_connections to 200+ in my.cnf:
  
  [!!] Highest connection usage: 100% (152/151) [!!] Aborted connections: 3.24% (330353/10202656)
  
  Right now this is the culprit that’s causing your DB error. Make sure you change it and then restart mysql.
  
  Reply Report

Answer 5

dwaynejaeger September 7, 2021

I had the same issue and solved it by adding a SWAP to my droplet following “How To Add Swap Space on Ubuntu 20.04”

How To Add Swap Space on Ubuntu 20.04

by Brian Boucheron

One way to guard against out-of-memory errors in applications is to add some swap space to your server. In this guide, we will cover how to add a swap file to an Ubuntu 20.04 server.

Reply Report

Related

Question

Having problem with Php fpm

Related

Leave a comment

Looking for something else?

Sign up for our newsletter

Related

Question

Having problem with Php fpm

Related

Leave a comment

Related

question

ERR_TOO_MANY_REDIRECTS

question

How to prevent bad caching on Users Browsers with Virtual Hosts on NGINX

question

Prosper 202 Install not working and getting an error

question

nginx welcome page home page

Looking for something else?