Having problem with Php fpm

October 2, 2018 2.4k views
Nginx Ubuntu 16.04
NewCustomer
By:
NewCustomer

Ok My site is working fine but when the number of visitors increases it breaks suddenly.To find out why it breaks I have been observing the behaviour of my server using the 'htop' command.

Using 'htop' I noticed that whenever the second value in the figure,

code Tasks 136 , 100 , 2 running

reaches 166 my site breaks,I get something like " 502 Gateway error" .The problem seems to be not coming from RAM because the RAM usage is just 4 to 5 GB but I have 16GB RAM.And also the CPU usage never reaches 100%.So where is the problem coming from? can anyone help me please!

My Server is information:

16GB RAM, 320 GB storage , 6 CPU

My NGinx configuration is :

worker_processes 6;
worker_connections 2000;

My PHP fpm configuration is :

pm = ondemand
pm.max_children = 100
pm.start_servers = 18
pm.min_spare_servers = 9
pm.max_spare_servers = 18
pm.process_idle_timeout = 10s;
pm.max_requests = 500

Am I missing something,I also use WP Super cache!!

Any help will be grateful,thank you in advance.

Log In to Comment
2 Answers
unixynet October 2, 2018

Increase max_children to 150 and restart fpm. Also, check the php-fpm error log. It should have further clues on why it's failing.

Reply
  • NewCustomer October 3, 2018

    I have increased the max_children to 150 but the problem still exists.Now instead of 166 my site breaks at 216.This does not mean my site can support more visitors than before.

    The max visitors supported when the maxchildren value was 100 is same as the max visitors supported when the maxchildren is set to 150. So no improvement in the performance of the site.

    Now the only thing that change are the first and second value in the figure :

    code Tasks 136 , 100 , 2 running

    to max value of 156 and 216 .Note there is still no improvement in the performance it is same as before.

    • unixynet October 3, 2018

      What's in the php fpmlog when the 502 error happens?

      • NewCustomer October 3, 2018

        Here is the log:

        [03-Oct-2018 13:09:34] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 10 idle, and 138 total children
[03-Oct-2018 13:09:35] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 10 idle, and 139 total children
[03-Oct-2018 13:09:36] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 6 idle, and 140 total children
[03-Oct-2018 13:09:46] WARNING: [pool www] server reached pm.max_children setting (150), consider raising it
[03-Oct-2018 13:17:31] WARNING: [pool www] server reached pm.max_children setting (150), consider raising it
        • unixynet October 3, 2018

          So you have two things to address

          1. You need to increase max_children again. Try 230
          2. Set start_servers to 50

          Try to see how much memory your php processes are consuming per run on average. So if it's 60MB on average, that tells you can increase maxchildren to a theoretical maximum of 16000/60 = 266. Your server will of course need to reserve RAM for other services. So perhaps 250 is your maximum maxchildren number.

        • unixynet October 5, 2018

          Check the logs again to see if there are any warnings like the ones above

          • NewCustomer October 7, 2018

            Now I am getting "Error establishing database connections" on reaching the maximum value of 215 and 216 for the first and second value in the figure:

            Tasks 215 , 216 , 2 running

          • unixynet October 7, 2018

            Make sure max_connections is set to 250 in your my.cnf. Then restart MySQL.

            Unlimited Servers. Unlimited Requests. https://www.unixy.net/server-management

          • NewCustomer October 10, 2018

            NO man it's still not working!!

            One thing I found out that whenever the tasks reaches max, in the front-end I get "Error establishing database connection" any idea why it is happening!!

unixynet October 10, 2018

It's happening because your max_connections is being reached. Run these commands and copy/paste the output here:

wget http://mysqltuner.pl
mv index.html mysqltuner.pl
perl mysqltuner.pl

On the last command, you might need to run it like this:

perl mysqltuner.pl --user root -pass <mysql_root_passwd>
Reply
  • NewCustomer October 10, 2018

    -------- Log file Recommendations ------------------------------------------------------------------
    [--] Log file: /var/log/mysql/error.log(106K)
    [OK] Log file /var/log/mysql/error.log exists
    [OK] Log file /var/log/mysql/error.log is readable.
    [OK] Log file /var/log/mysql/error.log is not empty
    [OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
    [OK] /var/log/mysql/error.log doesn't contain any warning.
    [!!] /var/log/mysql/error.log contains 291 error(s).
    [--] 0 start(s) detected in /var/log/mysql/error.log
    [--] 0 shutdown(s) detected in /var/log/mysql/error.log

    -------- Storage Engine Statistics -----------------------------------------------------------------
    [--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRGMYISAM +MyISAM +PERFORMANCESCHEMA
    [--] Data in InnoDB tables: 90.9M (Tables: 41)
    [OK] Total fragmented tables: 0

    -------- Analysis Performance Metrics --------------------------------------------------------------
    [--] innodbstatsonmetadata: OFF
    [OK] No stat updates during querying INFORMATION    SCHEMA.

    -------- Security Recommendations ------------------------------------------------------------------
    [OK] There are no anonymous accounts for any database users
    [OK] All database users have passwords assigned
    [!!] There is no basic password file list!

    -------- CVE Security Recommendations --------------------------------------------------------------
    [--] Skipped due to --cvefile option undefined

    -------- Performance Metrics -----------------------------------------------------------------------
    [--] Up for: 15d 5h 52m 5s (413M q [313.895 qps], 10M conn, TX: 13927G, RX: 46G)
    [--] Reads / Writes: 99% / 1%
    [--] Binary logging is disabled
    [--] Physical Memory : 15.7G
    [--] Max MySQL memory : 352.4M
    [--] Other process memory: 12.5G
    [--] Total buffers: 192.0M global + 1.1M per thread (151 max threads)
    [--] P_S Max memory usage: 72B
    [--] Galera GCache Max memory usage: 0B
    [OK] Maximum reached memory usage: 353.5M (2.20% of installed RAM)
    [OK] Maximum possible memory usage: 352.4M (2.20% of installed RAM)
    [OK] Overall possible memory usage with other process is compatible with memory available
    [OK] Slow queries: 0% (0/413M)
    [!!] Highest connection usage: 100% (152/151)
    [!!] Aborted connections: 3.24% (330353/10202656)
    [!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
    [!!] Query cache may be disabled by default due to mutex contention.
    [!!] Query cache efficiency: 0.0% (0 cached / 362M selects)
    [OK] Query cache prunes per day: 0
    [OK] Sorts requiring temporary tables: 1% (799K temp sorts / 51M sorts)
    [OK] No joins without indexes
    [!!] Temporary tables created on disk: 71% (14M on disk / 20M total)
    [OK] Thread cache hit rate: 97% (228K created / 10M connections)
    [OK] Table cache hit rate: 22% (398 open / 1K opened)
    [OK] Open file limit used: 0% (0/1K)
    [OK] Table locks acquired immediately: 100% (2K immediate / 2K locks)

    -------- Performance schema ------------------------------------------------------------------------
    [--] Memory used by P_S: 72B
    [--] Sys schema is installed.

    -------- ThreadPool Metrics ------------------------------------------------------------------------
    [--] ThreadPool stat is disabled.

    -------- MyISAM Metrics ----------------------------------------------------------------------------
    [!!] Key buffer used: 18.2% (3M used / 16M cache)
    [OK] Key buffer size / total MyISAM indexes: 16.0M/43.0K
    [!!] Read Key buffer hit rate: 85.2% (88 cached / 13 reads)
    [OK] Write Key buffer hit rate: 100.0% (10 cached / 10 writes)

    -------- InnoDB Metrics ----------------------------------------------------------------------------
    [--] InnoDB is enabled.
    [--] InnoDB Thread Concurrency: 0
    [OK] InnoDB File per table is activated
    [OK] InnoDB buffer pool / data size: 128.0M/90.9M
    [!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal 25%
    [OK] InnoDB buffer pool instances: 1
    [--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
    [OK] Innodbbufferpoolsize aligned with Innodbbufferpoolchunksize & Innodbbufferpoolinstances
    [OK] InnoDB Read buffer efficiency: 100.00% (48248427167 hits/ 48248428524 total)
    [!!] InnoDB Write Log efficiency: 10.35% (122327 hits/ 1181608 total)
    [OK] InnoDB log waits: 0.00% (0 waits / 1059281 writes)

    -------- AriaDB Metrics ----------------------------------------------------------------------------
    [--] AriaDB is disabled.

    -------- TokuDB Metrics ----------------------------------------------------------------------------
    [--] TokuDB is disabled.

    -------- XtraDB Metrics ----------------------------------------------------------------------------
    [--] XtraDB is disabled.

    -------- Galera Metrics ----------------------------------------------------------------------------
    [--] Galera is disabled.

    -------- Replication Metrics -----------------------------------------------------------------------
    [--] Galera Synchronous replication: NO
    [--] No replication slave(s) for this server.
    [--] Binlog format: ROW
    [--] XA support enabled: ON
    [--] Semi synchronous replication Master: Not Activated
    [--] Semi synchronous replication Slave: Not Activated
    [--] This is a standalone server

    -------- Recommendations ---------------------------------------------------------------------------
    General recommendations:
    Control error line(s) into /var/log/mysql/error.log file
    Reduce or eliminate persistent connections to reduce connection usage
    Reduce or eliminate unclosed connections and network issues
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    When making adjustments, make tmptablesize/maxheaptablesize equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Before changing innodb    logfilesize and/or innodblogfilesingroup read this: http://bit.ly/2wgkDvS
    Variables to adjust:
    maxconnections (> 151)
    wait    timeout (< 28800)
    interactivetimeout (< 28800)
    query    cachesize (=0)
    query    cachetype (=0)
    query    cachelimit (> 1M, or use smaller result sets)
    tmp    tablesize (> 16M)
    max    heaptablesize (> 16M)
    innodblogfile_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.

    • unixynet October 10, 2018

      I thought you changed max_connections to 200+ in my.cnf:

      [!!] Highest connection usage: 100% (152/151)
[!!] Aborted connections: 3.24% (330353/10202656)

      Right now this is the culprit that's causing your DB error. Make sure you change it and then restart mysql.

Have another answer? Share your knowledge.

Related Questions