Bug #62954 [Com]: startup problems fpm / php-fpm
Edit report at https://bugs.php.net/bug.php?id=62954&edit=1
ID: 62954
Comment by: per at techspot dot com
Reported by:jonas at brachium-system dot net
Summary:startup problems fpm / php-fpm
Status: Feedback
Type: Bug
Package:FPM related
Operating System: Linux
PHP Version:5.3Git-2012-08-27 (Git)
Assigned To:fat
Block user comment: N
Private report: N
New Comment:
This bug occurs more frequently on heavily loaded websites.
On my testserver it is difficult to reproduce with a low load.
But on my live server doing a "service php-fpm restart" makes it fail most
times.
For some reason the problem occurs less frequently in php-fpm 5.4.6 vs 5.3.16
opcode cachers also makes it occur more frequently, but it's possible to
reproduce with no opcode cacher loaded aswell, same rules as above for
frequency still apply (5.3 vs 5.4)
Output from testserver below running CentOS v6.3 with all patches applied and
php v5.4 via the remi repository:
http://rpms.famillecollet.com/enterprise/6/test/x86_64/
# php -v
PHP 5.4.6 (cli) (built: Aug 16 2012 14:26:39)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies
with XCache v2.0.1, Copyright (c) 2005-2012, by mOo
# DAEMON_COREFILE_LIMIT=unlimited strace -s1024 -f /etc/init.d/php-fpm restart
2>&1 | grep -i SEGV
[pid 3017] rt_sigaction(SIGSEGV, {0x7fddd5050730, [SEGV],
SA_RESTORER|SA_RESTART, 0x7fdddb55a920}, {SIG_DFL, [], 0}, 8) = 0
[pid 3017] rt_sigaction(SIGSEGV, {SIG_DFL, [SEGV], SA_RESTORER|SA_RESTART,
0x7fdddb55a920},
[pid 3017] <... rt_sigaction resumed> {0x7fddd5050730, [SEGV],
SA_RESTORER|SA_RESTART, 0x7fdddb55a920}, 8) = 0
Previous Comments:
[2012-09-02 09:07:29] jonas at brachium-system dot net
PHP 5.3.16-dev (fpm-fcgi) (built: Sep 2 2012 04:56:07)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2012 Zend Technologies
I have tried it with the current branch PHP-5.3.
It started two times correctly (Starts and detaches) after the second start the
problem occurs again.
So problem is not fixed - I have not tried 5.4 since fpm sapi is the same.
[2012-09-01 19:41:16] tony2...@php.net
I can't reproduce it here:
# /tmp/fpm/sbin/php-fpm -v
PHP 5.4.8-dev (fpm-fcgi) (built: Sep 1 2012 23:36:17) (DEBUG)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies
It starts, spawns its children and detaches from terminal with no problem at
all.
Can you try the latest snapshot and check if you still can reproduce the issue?
[2012-08-27 22:54:12] al-php at none dot at
same problem here.
./configure --prefix=// --enable-fpm --enable-bcmath --with-bz2
--enable-calendar --with-curl --enable-dba --enable-exif --enable-ftp
--enable-gd-native-ttf --with-jpeg-dir --with-png-dir --with-t1lib
--with-freetype-dir --with-gettext --enable-mbstring --with-mcrypt --with-mhash
--with-mysql=mysqlnd --with-mysqli=mysqlnd --with-pdo-mysql=mysqlnd
--enable-mysqlnd --with-openssl --with-pcre-regex --with-pspell --enable-shmop
--enable-soap --enable-sockets --enable-sysvmsg --enable-wddx --with-xsl
--enable-zip --with-zlib --with-tcadb --with-openssl-dir --with-gd
###
bin/php -v
PHP 5.4.5 (cli) (built: Aug 8 2012 01:03:00)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies
###
root@Ubuntu-1204-precise-64-minimal ~ # strace -fveall -p 2590 -p 2589
Process 2590 attached - interrupt to quit
Process 2589 attached - interrupt to quit
[pid 2589] wait4(-1,
[pid 2590] futex(0x7fc2a4d73720, FUTEX_WAIT_PRIVATE, 2, NULL
###
[2012-08-27 19:50:31] jonas at brachium-system dot net
Description:
PHP 5.3.14 with FPM on Debian Squeze works without any problems.
The PHP Version 5.3.15 / 5.3.16 and current Git branch PHP 5.3 is affected.
/usr/sbin/php5-fpm --fpm-config /etc/php5/fpm/php-fpm.conf
All pools are started and working, but the php5-fpm does not detach:
/usr/sbin/php5-fpm --fpm-config /etc/php5/fpm/php-fpm.conf
and here it hangs. ctrl +c does not work.
strace tells me:
Process 15949 attached - interrupt to quit
futex(0x7fb923e8ce40, FUTEX_WAIT_PRIVATE, 2, NULL
If I switch the configuration option "daemonize" to "no" php-fpm starts up
normal and can be aborted with ctrl+c.
If I copy sapi/fpm from 5.3.14 to the current php-src from git and compile it
again my configuration and php-fpm works and starts normal.
If I remove the USR1/USR2 handling from sapi/fpm/fpm/fpm_unix.c ( see patch )
php-fpm starts normal and d
Bug #62954 [Com]: startup problems fpm / php-fpm
Edit report at https://bugs.php.net/bug.php?id=62954&edit=1 ID: 62954 Comment by: per at techspot dot com Reported by:jonas at brachium-system dot net Summary:startup problems fpm / php-fpm Status: Feedback Type: Bug Package:FPM related Operating System: Linux PHP Version:5.3Git-2012-08-27 (Git) Assigned To:fat Block user comment: N Private report: N New Comment: Upgraded to PHP v5.4.7 but the issue remains: [root@dev01 ~]# service php-fpm stop Stopping php-fpm: [ OK ] [root@dev01 ~]# ps aux | grep php root 10912 0.0 0.0 103244 840 pts/0S+ 09:45 0:00 grep php [root@dev01 ~]# service php-fpm start Starting php-fpm: /bin/bash: line 1: 10923 Segmentation fault (core dumped) php-fpm [FAILED] [root@dev01 ~]# ps aux | grep php root 10924 0.0 0.1 516912 5096 ?Ss 09:45 0:00 php-fpm: master process (/etc/php-fpm.conf) nginx10926 0.0 0.1 516912 4256 ?S09:45 0:00 php-fpm: pool www nginx10927 0.0 0.1 516912 4256 ?S09:45 0:00 php-fpm: pool www nginx10928 0.0 0.1 516912 4256 ?S09:45 0:00 php-fpm: pool www nginx10929 0.0 0.1 516912 4256 ?S09:45 0:00 php-fpm: pool www nginx10930 0.0 0.1 516912 4256 ?S09:45 0:00 php-fpm: pool www /var/log/messages: Sep 13 09:45:51 dev01 kernel: php-fpm[10923]: segfault at 3b25051598 ip 003b1ca0eb83 sp 7fffe9cace00 error 4 in ld-2.12.so[3b1ca0+2] Sep 13 09:45:51 dev01 abrt[10925]: Saved core dump of pid 10923 (/usr/sbin/php-fpm) to /var/spool/abrt/ccpp-2012-09-13-09:45:51-10923 (5709824 bytes) Sep 13 09:45:51 dev01 abrtd: Directory 'ccpp-2012-09-13-09:45:51-10923' creation detected Sep 13 09:45:51 dev01 abrtd: Package 'php-fpm' isn't signed with proper key Sep 13 09:45:51 dev01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2012-09-13-09:45:51-10923' exited with 1 Sep 13 09:45:51 dev01 abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2012-09-13-09:45:51-10923, deleting Previous Comments: [2012-09-13 10:54:57] jonas at brachium-system dot net >This only shows us that FPM has successfully forked to detach from the console. >It didn't hang, crash or exit, it's still running and you can see it >yourself: > `ps afx | grep php` yep. every time I start php-fpm with gdb it forks and detach the process correctly. Then I start it without debugger it hangs. :( [2012-09-12 09:17:54] tony2...@php.net >Program received signal SIGUSR1, User defined signal 1 This only shows us that FPM has successfully forked to detach from the console. It didn't hang, crash or exit, it's still running and you can see it yourself: `ps afx | grep php` [2012-09-10 19:32:56] jonas at brachium-system dot net I have pulled the current PHP-5.3 branch and compile it with: ./configure --prefix=/usr --enable-fpm --enable-bcmath --with-bz2 --enable-calendar --with-curl --enable-dba --enable-exif --enable-ftp --enable-gd-native-ttf --with-jpeg-dir --with-png-dir --with-t1lib --with-freetype-dir --with-gettext --enable-mbstring --with-mcrypt --with-mhash --with-mysql=mysqlnd --with-mysqli=mysqlnd --with-pdo-mysql=mysqlnd --enable-mysqlnd --with-openssl --with-pcre-regex --with-pspell --enable-shmop --enable-soap --enable-sockets --enable-sysvmsg --enable-wddx --with-xsl --enable-zip --with-zlib --with-tcadb --with-openssl-dir --with-gd --enable-debug I have started PHP with gdb: - root@testm64:/home/jonas/php-src# gdb sapi/fpm/php-fpm GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /home/jonas/php-src/sapi/fpm/php-fpm...done. warning: not using untrusted file "/home/jonas/php-src/.gdbinit" (gdb) run Starting program: /home/jonas/php-src/sapi/fpm/php-fpm [Thread debugging using libthread_db enabled] Program received signal SIGUSR1, User defined signal 1. 0x749c8380 in ?? () from /lib/libc.so.6 (gdb) bt #0 0x749c8380 in ?? () from /lib/libc.so.6 #1 0x749cb94c i
Bug #62954 [Com]: startup problems fpm / php-fpm
Edit report at https://bugs.php.net/bug.php?id=62954&edit=1 ID: 62954 Comment by: per at techspot dot com Reported by:jonas at brachium-system dot net Summary:startup problems fpm / php-fpm Status: Feedback Type: Bug Package:FPM related Operating System: Linux PHP Version:5.3Git-2012-08-27 (Git) Assigned To:fat Block user comment: N Private report: N New Comment: Sorry Tony, the dumps are always deleted by all systems I have this problem on, see the last line of my paste before: "Corrupted or bad directory, deleting" Btw these bugs are probably the same as this one: #62886 #62937 Seems in #62886 a solution has been identified: "Anyway, we have a backtrace and the commit that introduced this issue." Previous Comments: [2012-09-17 14:39:46] tony2...@php.net jonas@ >Then I start it without debugger it hangs. :( then attach to the freezed process after that. just `gdb attach `, then `bt full` and post the result here. per@ Great, so you have a core file. Now type: gdb /usr/sbin/php-fpm /var/spool/abrt/ccpp-2012-09-13-09:45:51-10923 bt full and post the result here. ---- [2012-09-13 14:50:24] per at techspot dot com Upgraded to PHP v5.4.7 but the issue remains: [root@dev01 ~]# service php-fpm stop Stopping php-fpm: [ OK ] [root@dev01 ~]# ps aux | grep php root 10912 0.0 0.0 103244 840 pts/0S+ 09:45 0:00 grep php [root@dev01 ~]# service php-fpm start Starting php-fpm: /bin/bash: line 1: 10923 Segmentation fault (core dumped) php-fpm [FAILED] [root@dev01 ~]# ps aux | grep php root 10924 0.0 0.1 516912 5096 ?Ss 09:45 0:00 php-fpm: master process (/etc/php-fpm.conf) nginx10926 0.0 0.1 516912 4256 ?S09:45 0:00 php-fpm: pool www nginx10927 0.0 0.1 516912 4256 ?S09:45 0:00 php-fpm: pool www nginx10928 0.0 0.1 516912 4256 ?S09:45 0:00 php-fpm: pool www nginx10929 0.0 0.1 516912 4256 ?S09:45 0:00 php-fpm: pool www nginx10930 0.0 0.1 516912 4256 ?S09:45 0:00 php-fpm: pool www /var/log/messages: Sep 13 09:45:51 dev01 kernel: php-fpm[10923]: segfault at 3b25051598 ip 003b1ca0eb83 sp 7fffe9cace00 error 4 in ld-2.12.so[3b1ca0+2] Sep 13 09:45:51 dev01 abrt[10925]: Saved core dump of pid 10923 (/usr/sbin/php-fpm) to /var/spool/abrt/ccpp-2012-09-13-09:45:51-10923 (5709824 bytes) Sep 13 09:45:51 dev01 abrtd: Directory 'ccpp-2012-09-13-09:45:51-10923' creation detected Sep 13 09:45:51 dev01 abrtd: Package 'php-fpm' isn't signed with proper key Sep 13 09:45:51 dev01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2012-09-13-09:45:51-10923' exited with 1 Sep 13 09:45:51 dev01 abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2012-09-13-09:45:51-10923, deleting [2012-09-13 10:54:57] jonas at brachium-system dot net >This only shows us that FPM has successfully forked to detach from the console. >It didn't hang, crash or exit, it's still running and you can see it >yourself: > `ps afx | grep php` yep. every time I start php-fpm with gdb it forks and detach the process correctly. Then I start it without debugger it hangs. :( [2012-09-12 09:17:54] tony2...@php.net >Program received signal SIGUSR1, User defined signal 1 This only shows us that FPM has successfully forked to detach from the console. It didn't hang, crash or exit, it's still running and you can see it yourself: `ps afx | grep php` [2012-09-10 19:32:56] jonas at brachium-system dot net I have pulled the current PHP-5.3 branch and compile it with: ./configure --prefix=/usr --enable-fpm --enable-bcmath --with-bz2 --enable-calendar --with-curl --enable-dba --enable-exif --enable-ftp --enable-gd-native-ttf --with-jpeg-dir --with-png-dir --with-t1lib --with-freetype-dir --with-gettext --enable-mbstring --with-mcrypt --with-mhash --with-mysql=mysqlnd --with-mysqli=mysqlnd --with-pdo-mysql=mysqlnd --enable-mysqlnd --with-openssl --with-pcre-regex --with-pspell --enable-shmop --enable-soap --enable-sockets --enable-sysvmsg --enable-wddx --with-xsl --enable-zip --with-zlib --with-tcadb --with-openssl-dir --with-gd --enable-debug I have started PHP with gdb: - root@testm64:/home/jonas/php-src# gdb sapi/fpm/php-fpm GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009
Bug #62886 [Com]: PHP-FPM may segfault/hang on startup
Edit report at https://bugs.php.net/bug.php?id=62886&edit=1 ID: 62886 Comment by: per at techspot dot com Reported by:pierre at archlinux dot de Summary:PHP-FPM may segfault/hang on startup Status: Closed Type: Bug Package:FPM related Operating System: Arch Linux PHP Version:5.4.6 Assigned To:fat Block user comment: N Private report: N New Comment: fat: you can probably link this fix to bug #62937 aswell. APC is listed in that bug report, but any opcode cacher just makes this problem with php-fpm surface more regularly due to the increased concurrency... Thanks for the fix! (I'll wait for it's inclusion in the next PHP release though) Previous Comments: [2012-09-27 22:55:48] f...@php.net Automatic comment on behalf of fat Revision: http://git.php.net/?p=php-src.git;a=commit;h=75c63c5503b1c6b72e5e1daf5b4bfd02c68a4b79 Log: - Fixed bug #62954 (startup problems fpm / php-fpm) - Fixed bug #62886 (PHP-FPM may segfault/hang on startup) [2012-09-27 22:08:34] f...@php.net Automatic comment on behalf of fat Revision: http://git.php.net/?p=php-src.git;a=commit;h=b5eb1456aa17e83b7b734dfc2316d632c9adcc6c Log: - Fixed bug #62954 (startup problems fpm / php-fpm) - Fixed bug #62886 (PHP-FPM may segfault/hang on startup) [2012-09-27 22:08:32] f...@php.net Automatic comment on behalf of fat Revision: http://git.php.net/?p=php-src.git;a=commit;h=75c63c5503b1c6b72e5e1daf5b4bfd02c68a4b79 Log: - Fixed bug #62954 (startup problems fpm / php-fpm) - Fixed bug #62886 (PHP-FPM may segfault/hang on startup) [2012-09-27 18:51:10] jdr at alexanderinteractive dot com This patch seems to do the trick for me for php 5.4.7 Thanks! [2012-09-26 20:26:32] jonas at brachium-system dot net I came from #62954 - I have applied your patch against 5.3.x source. It fixes my problem - fpm starts up normal and detaches. fat, great work :) The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=62886 -- Edit this bug report at https://bugs.php.net/bug.php?id=62886&edit=1
Bug #62886 [Com]: PHP-FPM may segfault/hang on startup
Edit report at https://bugs.php.net/bug.php?id=62886&edit=1 ID: 62886 Comment by: per at techspot dot com Reported by:pierre at archlinux dot de Summary:PHP-FPM may segfault/hang on startup Status: Closed Type: Bug Package:FPM related Operating System: Arch Linux PHP Version:5.4.6 Assigned To:fat Block user comment: N Private report: N New Comment: Confirmed working on my testserver, thank you! The patch has already been backported into remi's repository, both v5.4 & v5.3 http://rpms.famillecollet.com/enterprise/6/remi/x86_64/repoview/php.html Previous Comments: [2012-09-29 05:35:51] f...@php.net done ! [2012-09-28 18:34:29] per at techspot dot com fat: you can probably link this fix to bug #62937 aswell. APC is listed in that bug report, but any opcode cacher just makes this problem with php-fpm surface more regularly due to the increased concurrency... Thanks for the fix! (I'll wait for it's inclusion in the next PHP release though) [2012-09-27 22:55:48] f...@php.net Automatic comment on behalf of fat Revision: http://git.php.net/?p=php-src.git;a=commit;h=75c63c5503b1c6b72e5e1daf5b4bfd02c68a4b79 Log: - Fixed bug #62954 (startup problems fpm / php-fpm) - Fixed bug #62886 (PHP-FPM may segfault/hang on startup) [2012-09-27 22:08:34] f...@php.net Automatic comment on behalf of fat Revision: http://git.php.net/?p=php-src.git;a=commit;h=b5eb1456aa17e83b7b734dfc2316d632c9adcc6c Log: - Fixed bug #62954 (startup problems fpm / php-fpm) - Fixed bug #62886 (PHP-FPM may segfault/hang on startup) [2012-09-27 22:08:32] f...@php.net Automatic comment on behalf of fat Revision: http://git.php.net/?p=php-src.git;a=commit;h=75c63c5503b1c6b72e5e1daf5b4bfd02c68a4b79 Log: - Fixed bug #62954 (startup problems fpm / php-fpm) - Fixed bug #62886 (PHP-FPM may segfault/hang on startup) The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=62886 -- Edit this bug report at https://bugs.php.net/bug.php?id=62886&edit=1
