Apache2 Php-Fpm No Response on Same Time Daily (504 Gateway Timeout)

September 19, 2019 359 views
Apache PHP

Hi,

My server stop response everyday at the same time (12.15pm-12.30pm),
I already check with my server cron job but did not see any cron are working in that time.
I also make some change to my mpm event file to increase the MaxRequestWorkers but the problem still unsolved.
Do you have any suggestion for me the fix this?

Below is my error.log

[Tue Sep 17 11:48:59.874938 2019] [mpm_event:notice] [pid 27614:tid 139782794283968] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1c configured -- resuming normal operations
[Tue Sep 17 11:48:59.874997 2019] [core:notice] [pid 27614:tid 139782794283968] AH00094: Command line: '/usr/sbin/apache2'
[Tue Sep 17 12:02:11.049009 2019] [mpm_event:notice] [pid 27614:tid 139782794283968] AH00491: caught SIGTERM, shutting down
[Tue Sep 17 12:02:11.176826 2019] [mpm_event:notice] [pid 27898:tid 140114637085632] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1c configured -- resuming normal operations
[Tue Sep 17 12:02:11.176900 2019] [core:notice] [pid 27898:tid 140114637085632] AH00094: Command line: '/usr/sbin/apache2'
[Tue Sep 17 12:09:26.041604 2019] [proxy_fcgi:error] [pid 27900:tid 140114088937216] [client 54.36.150.68:22292] AH01071: Got error 'PHP message: PHP Warning:  sizeof(): Parameter must be an array or an object that implements Countable in /var/www/babyneeds/html/wp-content/plugins/woocommerce-brands/includes/class-wc-brands.php on line 536'
[Tue Sep 17 12:12:36.789535 2019] [mpm_event:error] [pid 27898:tid 140114637085632] AH10159: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting
[Tue Sep 17 12:12:38.794284 2019] [mpm_event:error] [pid 27898:tid 140114637085632] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Tue Sep 17 12:17:21.253383 2019] [proxy_fcgi:error] [pid 28375:tid 140114013402880] (70007)The timeout specified has expired: [client 18.140.116.218:59872] AH01075: Error dispatching request to : (polling)
[Tue Sep 17 12:17:21.527966 2019] [proxy_fcgi:error] [pid 28308:tid 140114030188288] (70007)The timeout specified has expired: [client 13.251.125.164:38144] AH01075: Error dispatching request to : (polling)
[Tue Sep 17 12:17:21.661439 2019] [proxy_fcgi:error] [pid 28375:tid 140113937819392] (70007)The timeout specified has expired: [client 13.229.147.238:36646] AH01075: Error dispatching request to : (polling)
[Tue Sep 17 12:17:21.662047 2019] [proxy_fcgi:error] [pid 28308:tid 140114114115328] (70007)The timeout specified has expired: [client 18.138.102.29:54436] AH01075: Error dispatching request to : (polling)
ue Sep 17 12:24:41.069317 2019] [proxy_fcgi:error] [pid 27899:tid 140114021795584] (70007)The timeout specified has expired: [client 66.249.71.94:62906] AH01075: Error dispatching request to : (polling)
[Tue Sep 17 12:25:01.507350 2019] [proxy_fcgi:error] [pid 28441:tid 140114038580992] (70007)The timeout specified has expired: [client 157.245.196.201:42998] AH01075: Error dispatching request to : (polling)
[Tue Sep 17 12:25:01.684032 2019] [proxy_fcgi:error] [pid 28441:tid 140114223212288] (70007)The timeout specified has expired: [client 10.15.0.2:37508] AH01075: Error dispatching request to : (polling)
[Tue Sep 17 12:25:19.403728 2019] [proxy_fcgi:error] [pid 28441:tid 140113996617472] (70007)The timeout specified has expired: [client 46.229.168.140:4238] AH01075: Error dispatching request to : (polling)
[Tue Sep 17 12:25:59.837218 2019] [proxy_fcgi:error] [pid 28308:tid 140114055366400] (70007)The timeout specified has expired: [client 54.36.148.133:35380] AH01075: Error dispatching request to : (polling)
[Tue Sep 17 12:53:55.160605 2019] [mpm_event:notice] [pid 27898:tid 140114637085632] AH00491: caught SIGTERM, shutting down
[Tue Sep 17 12:53:55.260282 2019] [mpm_event:notice] [pid 29414:tid 140194432011200] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1c configured -- resuming normal operations
[Tue Sep 17 12:53:55.260357 2019] [core:notice] [pid 29414:tid 140194432011200] AH00094: Command line: '/usr/sbin/apache2'
[Tue Sep 17 13:02:06.660028 2019] [mpm_event:notice] [pid 29414:tid 140194432011200] AH00491: caught SIGTERM, shutting down
[Tue Sep 17 13:02:06.785821 2019] [mpm_event:notice] [pid 30108:tid 140469778623424] AH00489: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1c configured -- resuming normal operations
[Tue Sep 17 13:02:06.785878 2019] [core:notice] [pid 30108:tid 140469778623424] AH00094: Command line: '/usr/sbin/apache2'

Thank you.

1 Answer

Hi @ZhiJie,

According to the logs, it does seem like the ApacheWorkers have hit their limit and afterwards some requests have timed out due to the wait time.

Increasing the ApacheWorkers was a good solution however it it’s still happening you’ll need to further investigate. I have two suggestions, one is during the said time, someone is on purpose flooding your server. The other one is your website gets updated or something similar and there are a lot of requests which actually make the server unusable.

Anyway, to actually confirm this, you can try and run the following command and see what is feeling up your ApacheWorkers

apachectl fullstatus

Or if this doesn’t work

service httpd fullstatus
service apache2 fullstatus

I’ve had cases where apps created a million connections at a specific time and didn’t close them up afterwards causing my server to reach it’s maximum.

Kind regards,
Kalin

  • Hi @KFerdinandov , thank you for your suggestion

    When the server downtime today, i try to check apachectl fullstatus & my server response as code below :

    Gateway Timeout
    
    The gateway did not receive a timely response from upstream server or application.
    
    ------------------------------------------------------------------------------
    Apache/2.4.41 (Ubuntu) Server at localhost Port 80
    

    It’s really weird because after the 15 minutes downtime everyday,
    the server are back to normal without any apache or php-fpm restart until the next day with the same time the server will down again.
    When the server down, i don’t see any activities or CPU & MEM overload at the time.

    Attached is “TOP” for your reference
    Imgur

    & Htop
    Imgur

    Looking forward for your suggestion.

    Thank you.

Have another answer? Share your knowledge.