php-fpm7.2 segfault

April 22, 2019 613 views
PHP Ubuntu 18.04

Server Details:

ubuntu 18.04.02 LTS
nginx 1.14.0
php7.2-fpm
4GB ram machine

I keep get 50x errors in my website and the only way so far to bring it back to normal is to restart php7.2-fpm service.

Logs:

php7.2-fpm.log

[21-Apr-2019 14:02:34] WARNING: [pool mywebsite.gr] child 23244 exited on signal 11 (SIGSEGV - core dumped) after 38.322120 seconds from start
[21-Apr-2019 14:02:34] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:35] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:35] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:35] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:39] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:40] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:40] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:40] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:42] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:42] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:43] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:43] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:44] WARNING: failed processes threshold (10 in 60 sec) is reached, initiating reload
[21-Apr-2019 14:02:44] NOTICE: reloading: execvp("/usr/sbin/php-fpm7.2", {"/usr/sbin/php-fpm7.2", "--nodaemonize", "--fpm-config", "/etc/php/7.2/fpm/php-fpm.conf"})
[21-Apr-2019 14:29:06] NOTICE: fpm is running, pid 23509
[21-Apr-2019 14:29:06] NOTICE: ready to handle connections
[21-Apr-2019 14:29:06] NOTICE: systemd monitor interval set to 10000ms
[21-Apr-2019 14:37:42] WARNING: [pool mywebsite.gr] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 14 idle, and 26 total children

kern.log

Apr 22 09:19:53 website-18-04-2vcpu-4gb-sfo2-01 kernel: [55794.545022] php-fpm7.2[9540]: segfault at 5569c31565f0 ip 00005569c262d0e3 sp 00007ffc3ef36450 error 4 in php-fpm7.2[5569c2383000+415000]

nginx.error.log

2019/04/21 17:40:50 [alert] 21615#21615: aborting
malloc(): memory corruption
2019/04/21 20:08:47 [alert] 1023#1023: worker process 1025 exited on signal 6 (core dumped)
2019/04/21 22:46:08 [alert] 1023#1023: worker process 3841 exited on signal 11 (core dumped)
2019/04/21 22:51:14 [alert] 1023#1023: worker process 5010 exited on signal 11 (core dumped)
2019/04/21 22:55:59 [alert] 1023#1023: worker process 5023 exited on signal 11 (core dumped)
2019/04/22 00:05:04 [alert] 1023#1023: worker process 5030 exited on signal 11 (core dumped)
2019/04/22 00:43:12 [alert] 1023#1023: worker process 5503 exited on signal 11 (core dumped)

Configurations

/etc/php/7.2/fpm/pool.d/mywebsite.gr.conf

[mywebsite.gr]
user = wwwwebsite
group = wwwwebsite
listen = /var/run/php/php7.2-fpm-mywebsite.gr.sock
listen.owner = www-data
listen.group = www-data
php_admin_value[disable_functions] = exec,passthru,shell_exec,system
php_admin_flag[allow_url_fopen] = on
pm = dynamic
pm.max_children = 40
pm.start_servers = 15
pm.min_spare_servers = 15
pm.max_spare_servers = 25
pm.max_requests = 250
pm.process_idle_timeout = 10s
chdir = /
request_terminate_timeout = 120s

/etc/nginx/sites-available/mywebsite.gr

server {
        root /var/www/mywebsite.gr/wwwroot;
        index index.php index.html index.htm;

        server_name mywebsite.gr;

        error_log /var/www/mywebsite.gr/logs/mywebsite_error.log;
        access_log /var/www/mywebsite.gr/logs/mywebsite_access.log;

        client_max_body_size 64M;

        location / {
                try_files $uri $uri/ /index.php?q=$uri&$args;
        }

        error_page 404 /404.html;

        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
                root /usr/share/nginx/html;
        }

        location ~ /.well-known {
                allow all;
        }

        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                fastcgi_pass unix:/var/run/php/php7.2-fpm-mywebsite.gr.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_param SCRIPT_NAME $fastcgi_script_name;
                fastcgi_read_timeout 120;
                include fastcgi_params;
        }
}

Finally, the command

dmesg | grep segfault | tail -10

outputs:

[34431.289597] php-fpm7.2[6853]: segfault at 556f68626190 ip 0000556f65fe10e3 sp 00007fff4f168520 error 4 in php-fpm7.2[556f65d37000+415000]
[34432.372727] php-fpm7.2[4736]: segfault at 556f686261d8 ip 00007f8ba3796f81 sp 00007fff4f16a790 error 4 in opcache.so[7f8ba378b000+65000]
[44650.082264] systemd-udevd[5259]: segfault at 559d1cc11e50 ip 0000559d1cc11e50 sp 00007ffeb00bf5c0 error 15
[49936.341425] sshd[8670]: segfault at 0 ip 0000000000000000 sp 00007ffd8a189520 error 14 in sshd[5613a7e2e000+bd000]
[52274.639004] php-fpm7.2[8923]: segfault at 7fbc1a0d5000 ip 00007fbc1a0d5000 sp 00007ffc3ef36220 error 15
[52779.065779] apport[9060]: segfault at 7f878ee7faf0 ip 00007f878ee7faf0 sp 00007ffcf4540678 error 14
[53738.698410] php-fpm7.2[8965]: segfault at 7fbc1c980008 ip 00007fbc1c980008 sp 00007ffc3ef36190 error 15 in zero (deleted)[7fbc1b44d000+8000000]
[53760.422003] php-fpm7.2[9201]: segfault at 7fbc193dfa85 ip 00005569c2657c48 sp 00007ffc3ef37f30 error 4 in php-fpm7.2[5569c2383000+415000]
[54022.130290] php-fpm7.2[9233]: segfault at 1 ip 00007fbc34b9fb40 sp 00007ffc3ef36190 error 6 in libpcre.so.3.13.3[7fbc34b35000+70000]
[54756.617259] php-fpm7.2[9218]: segfault at 5569c30af420 ip 00005569c30af420 sp 00007ffc3ef36550 error 15
1 Answer

Hello,

The first thing that I could suggest is to try and update your PHP 7.2 minor version to the latest one.

If this does not help, then you would need to enable PHP-FPM core dumps and then use a tool like gdb to read the core dumps for more information.

To enable the Linux core dumps, you could use this command here:

echo '/tmp/coredump-%e.%p' > /proc/sys/kernel/core_pattern

This means that whenever there is a segmentation fault a core dump would be saved at /tmp/coredump-*.

Then the gdb command would allow you to read the coredump:

gdb /usr/sbin/php-fpm /tmp/coredump-php-fpm.2393

After that once the gdb has loaded the coredump, type bt to get the backtrace. In the backtrace you would have all of the information on what's causing the segmentation faults.

Hope that this helps!
Regards,
Bobby

Have another answer? Share your knowledge.