How do you fix a php5-fpm kernel segfault error?

December 20, 2015 2k views
LEMP PHP Ubuntu

A couple of days ago, my OSSEC daemon started sending me syslogs about a segfault in my php5-fpm. Here's the portion of the logs:

Dec 19 15:09:38 mydropplet kernel: [165740.998932] php5-fpm[26936]: segfault at 7fff621ad001 ip 000000000079fe74 sp 00007fff621a5040 error 6 in php5-fpm[400000+800000]

So I peeked into my error logs to see if I had some PHP errors somewhere that were causing the segfault (since I am not sure what that even means, but sounds like it's something in the core) My error logs had this in it:

2015/12/19 01:36:49 [error] 6177#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 01:36:54 [error] 6177#0: gv.symcd.com could not be resolved (110: Operation timed out) while requesting certificate status, responder: gv.symcd.com
2015/12/19 02:00:06 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:11 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:16 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:21 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:26 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx
2015/12/19 02:00:31 [error] 6178#0: recv() failed (111: Connection refused) while resolving, resolver: xxx.xxx.xxx.xxx:xx

So I took a closer look at my syslogs it had this in it:

(segfault error)

Dec 19 09:44:06 mydomain kernel: [146209.162601] php5-fpm[13787]: segfault at 7fff621ad001 ip 000000000079fe74 sp 00007fff621a5040 error 6 in php5-fpm[400000+800000]

(the rest of it)

Dec 19 09:44:59 mydomain postfix/anvil[26420]: statistics: max connection rate 1/60s for (smtp:195.22.126.137) at Dec 19 09:41:39
Dec 19 09:44:59 mydomain postfix/anvil[26420]: statistics: max connection count 1 for (smtp:195.22.126.137) at Dec 19 09:41:39
Dec 19 09:44:59 mydomain  postfix/anvil[26420]: statistics: max cache size 1 at Dec 19 09:41:39
Dec 19 09:40:01 mydomain  CRON[26308]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 > /dev/null | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
Dec 19 09:40:36 mydomain kernel: [145998.783586] [UFW BLOCK] IN=eth0 OUT= MAC= [...] SRC=208.69.30.108 DST=[...] LEN=40 TOS=0x08 PREC=0x00 TTL=[...] ID=44687 PROTO=TCP SPT=42987 DPT=9999 WINDOW=1024 RES=0x00 SYN URGP=0 
Dec 19 09:41:01 mydomain CRON[26376]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 > /dev/null | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
Dec 19 09:41:39 mydomain postfix/smtpd[26418]: connect from unknown[195.22.126.137]
Dec 19 09:41:39 mydomain postfix/smtpd[26418]: disconnect from unknown[195.22.126.137]

I'm pretty new to self-managed servers, so I am not sure how to debug this one. I manage my personal website on this server and the only thing I did a couple days ago is installed a new plugin from the Wordpress repository, due to lack of time to research a proper configuration I hastily activated it and configured it, and it bricked the front end of my site. So I deactivated it, and decided to give it a go later on. After that the kernel errors started happening. So I uninstalled the plugin completely thinking it might be the cause, but it didn't solve anything. The plugin would have only had write access to my web directory and would not have been able to change any core configuration settings. I'm not sure if there's a connection there, but I thought I would mention it.

I have also been running this configuration since October with no problems. Before seeing the error I executed a dist-upgrade as a part of my regular maintenance:

sudo apt-get dist-upgrade
sudo apt-get update
sudo shutdown -r now

This was around the same time as the plugin installation. So I thought I'd just execute the commands again, thinking if there were a bug surely a patch would have been pushed. Basically I was trying to knock around any bugs introduced from the last time I updated. This, however, didn't help. So I'm at a loss of what I can do to beat this.

Any advice would be great! I'm running a LEMP Ubuntu 14.04 LTS server image and have images available to roll back to, if needed, but not something I'm hankering to do.

Be the first one to answer this question.