mpruce
By:
mpruce

Apache2 crashing - segmentation fault

February 23, 2015 4.5k views

I submitted this as a support ticket but was told it is outside of their scope to help me. Any help here is appreciated, I'll buy you a beer or something if you figure this out with me!

Apache2 is intermittently crashing on my Ubuntu server and I've done a lot of debugging but can't find the source of the issue.

Details:

  • When the error occurs, my site is down, no output or error message. In /var/log/apache2/error.log I get messages that look like this:
[core:notice] [pid 6113] AH00051: child pid 9744 exit signal Segmentation fault (11), possible coredump in /tmp/apache2-gdb-dump 
  • I set up that gdb dump directory and set apache to dump its core before crashing, below are the contents of that file:
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).

Core was generated by `/usr/sbin/apache2 -k start'. 
Program terminated with signal SIGSEGV, Segmentation fault. 
<^>#0 0x00007f40b4b86091 in _zend_hash_quick_add_or_update (ht=ht@entry=0x7f40b533d6f0 <executor_globals+432>, 
arKey=0x7f40a05366f0 "/var/www/buydontrent.com/public_html/prod/index.php", nKeyLength=52, h=5705093358713398062, 
pData=pData@entry=0x7fffa37dd408, nDataSize=nDataSize@entry=8, pDest=pDest@entry=0x0, flag=flag@entry=2) 
at /build/buildd/php5-5.5.9+dfsg/Zend/zend_hash.c:279 
279 /build/buildd/php5-5.5.9+dfsg/Zend/zend_hash.c: No such file or directory.<^> 

I assume the warnings are not important, the last few line are probably the culprits. I haven't even been able to find those files because /build/ is not a directory in my root, so I don't know where to look. Plus I don't know C so I'm not sure if I would be able to figure out the problem anyway.

I've searched the dregs of the internet for this issue and found one DigitalOcean thread in which users concluded to try a new hosting company (and had success) and some mention of OPCache in another thread, but to my knowledge that isn't being used on my set up.

Thanks for any help,
Mark

3 comments
  • Doing a bit of looking into this it sounds like Apache segmentation faults can sometimes be caused by Zend when the process runs out of memory. I would recommend checking available memory while apache is running by running free -m and setting up a swap file which should prevent out of memory crashes.

    While I have not encountered this particular issue before this is where I would start. if this fails to resolve the issue let us know here.

    by Justin Ellingwood
    Swap space can be used as an "overflow" area for your system when you run out of RAM. The operating system can store data that would normally be kept in RAM on the hard drive in a specially formatted file. In this guide, we'll demonstrate how to create and use one of these files in Ubuntu 14.04.
  • Thanks! After the last crash I increased the size of my droplet just in case this was memory related, I've also increased the php memory limit to 768. Since the droplet size increase, apache hasn't crashed but it hasn't been that long so its too early to say whether it helped. I will look into the swap file link as well. Thanks so much.

  • A day later and no crash, so this must have been caused by lack of memory - thanks for pointing me in the right direction!

1 Answer

This question was answered by @ryanpq:

Doing a bit of looking into this it sounds like Apache segmentation faults can sometimes be caused by Zend when the process runs out of memory. I would recommend checking available memory while apache is running by running free -m and setting up a swap file which should prevent out of memory crashes.

While I have not encountered this particular issue before this is where I would start. if this fails to resolve the issue let us know here.

View the original comment

by Justin Ellingwood
Swap space can be used as an "overflow" area for your system when you run out of RAM. The operating system can store data that would normally be kept in RAM on the hard drive in a specially formatted file. In this guide, we'll demonstrate how to create and use one of these files in Ubuntu 14.04.
Have another answer? Share your knowledge.