romanaft
By:
romanaft

Apache goes down every few days

March 15, 2018 357 views
Apache MySQL Ubuntu 16.04

Hello all,
I'm trying to find a solution why apache stops constantly about every week. Sometimes mysql also and the need a restart to work again. I use Ubuntu Linux 16.04.3, Webmin version 1.872 and Virtualmin version 6.02.
My droplet has 2 VCPU, 4GB RAM, 80 Hard Drive.
My installation is clean, I have not added or changed anything. The most extreme thing I did is to increase space on digital ocean. What

I can see on apache error log is:

Mon Mar 12 04:51:26.202652 2018] [fcgid:warn] [pid 10923:tid 139857786857344] modfcgid: process 13273 graceful kill fail, sending SIGKILL [Mon Mar 12 05:12:51.575849 2018] [fcgid:warn] [pid 10923:tid 139857786857344] modfcgid: process 14964 graceful kill fail, sending SIGKILL [Mon Mar 12 05:24:07.158648 2018] [fcgid:warn] [pid 10923:tid 139857786857344] modfcgid: process 13271 graceful kill fail, sending SIGKILL [Mon Mar 12 05:41:21.174643 2018] [fcgid:warn] [pid 10923:tid 139857786857344] modfcgid: process 16809 graceful kill fail, sending SIGKILL [Mon Mar 12 05:46:43.061688 2018] [fcgid:warn] [pid 10923:tid 139857786857344] modfcgid: process 17155 graceful kill fail, sending SIGKILL [Mon Mar 12 05:51:29.944181 2018] [fcgid:warn] [pid 10923:tid 139857786857344] modfcgid: process 17521 graceful kill fail, sending SIGKILL [Mon Mar 12 05:57:19.267152 2018] [fcgid:warn] [pid 10923:tid 139857786857344] modfcgid: process 18140 graceful kill fail, sending SIGKILL [Mon Mar 12 06:07:35.777688 2018] [fcgid:warn] [pid 10923:tid 139857786857344] modfcgid: process 18907 graceful kill fail, sending SIGKILL [Mon Mar 12 06:25:10.350421 2018] [mpmevent:notice] [pid 1831:tid 139857786857344] AH00493: SIGUSR1 received. Doing graceful restart [Mon Mar 12 06:25:10.659935 2018] [core:notice] [pid 1831] AH00060: seg fault or similar nasty error detected in the parent process [Mon Mar 12 07:18:59.580713 2018] [suexec:notice] [pid 24254:tid 140281696900992] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec) [Mon Mar 12 07:18:59.598500 2018] [auth_digest:notice] [pid 24255:tid 140281696900992] AH01757: generating secret for digest authentication ... [Mon Mar 12 07:18:59.605701 2018] [core:warn] [pid 24255:tid 140281696900992] AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown of previous Apache run? [Mon Mar 12 07:18:59.606791 2018] [mpm_event:notice] [pid 24255:tid 140281696900992] AH00489: Apache/2.4.18 (Ubuntu) modfcgid/2.3.9 OpenSSL/1.0.2g configured -- resuming normal operations [Mon Mar 12 07:18:59.606809 2018] [core:notice] [pid 24255:tid 140281696900992] AH00094: Command line: '/usr/sbin/apache2' [Mon Mar 12 07:19:26.429963 2018] [core:warn] [pid 24255:tid 140281696900992] AH00045: child process 24261 still did not exit, sending a SIGTERM [Mon Mar 12 07:19:28.431618 2018] [core:warn] [pid 24255:tid 140281696900992] AH00045: child process 24261 still did not exit, sending a SIGTERM [Mon Mar 12 07:19:30.433796 2018] [core:warn] [pid 24255:tid 140281696900992] AH00045: child process 24261 still did not exit, sending a SIGTERM [Mon Mar 12 07:19:32.435943 2018] [core:error] [pid 24255:tid 140281696900992] AH00046: child process 24261 still did not exit, sending a SIGKILL [Mon Mar 12 07:19:33.437141 2018] [mpmevent:notice] [pid 24255:tid 140281696900992] AH00491: caught SIGTERM, shutting down [Mon Mar 12 07:19:34.450660 2018] [fcgid:error] [pid 24258:tid 140281696900992] FastCGI process 24501 still did not exit, terminating forcefully [Mon Mar 12 07:19:36.264629 2018] [suexec:notice] [pid 24695:tid 140038449141632] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec) [Mon Mar 12 07:19:36.282998 2018] [auth_digest:notice] [pid 24696:tid 140038449141632] AH01757: generating secret for digest authentication ... [Mon Mar 12 07:19:36.291232 2018] [mpm_event:notice] [pid 24696:tid 140038449141632] AH00489: Apache/2.4.18 (Ubuntu) modfcgid/2.3.9 OpenSSL/1.0.2g configured -- resuming normal operations [Mon Mar 12 07:19:36.291253 2018] [core:notice] [pid 24696:tid 140038449141632] AH00094: Command line: '/usr/sbin/apache2' [Mon Mar 12 07:24:57.965652 2018] [fcgid:warn] [pid 24699:tid 140038449141632] modfcgid: process 26560 graceful kill fail, sending SIGKILL [Mon Mar 12 07:27:26.086554 2018] [fcgid:warn] [pid 24699:tid 140038449141632] modfcgid: process 25069 graceful kill fail, sending SIGKILL

It is a repeated pattern in my apache error log. I don't know what all these mean and it would be great to show me some directions! Thank you in advance.

3 Answers

Have you looked into your access logs on the server to see what kinda traffic your droplet is getting and perhaps this is the result of a DDoS?

  • Hello I can not recognise a DDos my I see something strange at syslog.

    Mar 16 07:14:40 ns3 postfix/smtpd[20870]: warning: hostname walkerj235.com does not resolve to address 91.200.12.166
    Mar 16 07:14:42 ns3 postfix/smtpd[20870]: warning: unknown[91.200.12.166]: SASL LOGIN authentication failed: authentication failure
    Mar 16 07:19:22 ns3 postfix/smtpd[23304]: warning: www.prepa2.unam.mx[132.248.92.218]: SASL LOGIN authentication failed: authentication failure
    Mar 16 07:19:48 ns3 postfix/smtpd[23304]: warning: www.prepa2.unam.mx[132.248.92.218]: SASL LOGIN authentication failed: authentication failure
    Mar 16 07:21:10 ns3 postfix/smtpd[23304]: warning: unknown[91.234.99.214]: SASL LOGIN authentication failed: authentication failure
    Mar 16 07:23:31 ns3 postfix/smtpd[24618]: warning: hostname walkerj235.com does not resolve to address 91.200.12.166
    Mar 16 07:23:34 ns3 postfix/smtpd[24618]: warning: unknown[91.200.12.166]: SASL LOGIN authentication failed: authentication failure

    Is someone tryings to sends emails from my server? What the first line means? I can not recognise any of the domains above.

Today apache failed to start again as it crashed again. Digging in log files I discover that it is happening every sunday at 06:25! It crashes every week at the same time after a few graceful restarts with the error: AH00060: seg fault or similar nasty error detected in the parent process.
Searching for that error returns results regarding log rotation issues, apache modules causing issue
The logs during crash:

Corrupt JPEG data: 148 extraneous bytes before marker 0xb7
Unsupported marker type 0xb7
Corrupt JPEG data: 808 extraneous bytes before marker 0x13
Unsupported marker type 0x13
Corrupt JPEG data: 301 extraneous bytes before marker 0x6f
Unsupported marker type 0x6f
Corrupt JPEG data: 117 extraneous bytes before marker 0xd5
Unsupported marker type 0x38
Corrupt JPEG data: 143 extraneous bytes before marker 0x37
Unsupported marker type 0x37
Corrupt JPEG data: 253 extraneous bytes before marker 0x10
Unsupported marker type 0x10
Corrupt JPEG data: 289 extraneous bytes before marker 0x16
Unsupported marker type 0x16
[Sat Mar 17 06:25:11.000097 2018] [mpm_event:notice] [pid 24696:tid 140038449141632] AH00493: SIGUSR1 received.  Doing graceful restart
[Sat Mar 17 06:25:12.597654 2018] [auth_digest:notice] [pid 24696:tid 140038449141632] AH01757: generating secret for digest authentication ...
[Sat Mar 17 06:25:12.639362 2018] [mpm_event:notice] [pid 24696:tid 140038449141632] AH00489: Apache/2.4.18 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/1.0.2g configured -- resuming normal operations
[Sat Mar 17 06:25:12.639535 2018] [core:notice] [pid 24696:tid 140038449141632] AH00094: Command line: '/usr/sbin/apache2'
[Sat Mar 17 08:41:32.832883 2018] [mpm_event:notice] [pid 24696:tid 140038449141632] AH00493: SIGUSR1 received.  Doing graceful restart
[Sat Mar 17 08:41:36.215165 2018] [auth_digest:notice] [pid 24696:tid 140038449141632] AH01757: generating secret for digest authentication ...
[Sat Mar 17 08:41:36.228798 2018] [mpm_event:notice] [pid 24696:tid 140038449141632] AH00489: Apache/2.4.18 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/1.0.2g configured -- resuming normal operations
[Sat Mar 17 08:41:36.228862 2018] [core:notice] [pid 24696:tid 140038449141632] AH00094: Command line: '/usr/sbin/apache2'
[Sat Mar 17 08:46:03.264201 2018] [mpm_event:notice] [pid 24696:tid 140038449141632] AH00493: SIGUSR1 received.  Doing graceful restart
[Sat Mar 17 08:46:06.369497 2018] [auth_digest:notice] [pid 24696:tid 140038449141632] AH01757: generating secret for digest authentication ...
[Sat Mar 17 08:46:06.376898 2018] [mpm_event:notice] [pid 24696:tid 140038449141632] AH00489: Apache/2.4.18 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/1.0.2g configured -- resuming normal operations
[Sat Mar 17 08:46:06.376923 2018] [core:notice] [pid 24696:tid 140038449141632] AH00094: Command line: '/usr/sbin/apache2'
[Sun Mar 18 06:25:10.960233 2018] [mpm_event:notice] [pid 24696:tid 140038449141632] AH00493: SIGUSR1 received.  Doing graceful restart
[Sun Mar 18 06:25:11.212430 2018] [core:notice] [pid 24696] AH00060: seg fault or similar nasty error detected in the parent process
[Sun Mar 18 08:40:37.709116 2018] [suexec:notice] [pid 12228:tid 140325802383232] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Sun Mar 18 08:40:37.740299 2018] [auth_digest:notice] [pid 12230:tid 140325802383232] AH01757: generating secret for digest authentication ...
[Sun Mar 18 08:40:37.752820 2018] [core:warn] [pid 12230:tid 140325802383232] AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Sun Mar 18 08:40:37.754032 2018] [mpm_event:notice] [pid 12230:tid 140325802383232] AH00489: Apache/2.4.18 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/1.0.2g configured -- resuming normal operations
[Sun Mar 18 08:40:37.754056 2018] [core:notice] [pid 12230:tid 140325802383232] AH00094: Command line: '/usr/sbin/apache2'
Have another answer? Share your knowledge.