Hi all,
we have an Oracle Solaris 11.3 x86_64-based server with the following
components:
- Apache 2.4.12(-0.175.3.0.0.30.0)
- SVN 1.7.20(-5.12.0.0.0.91.0)
SVN is connected to Apache with "mod_dav_svn" and I have a repository with a
"post-commit" hook.
The Apache package is the original package with comes with Solaris 11.3, while
SVN is compiled on a Solaris build host by ourselves because Oracle missed to
include 64 bit binaries in Solaris 11.3 and they are needed when one tries to
include "mod_dav_svn.so" into a 64 bit version of the Apache web server, of
course. But this shouldn't be the cause on any problem.
My problem is: The whole configuration is running absolutely fine as long as
Apache is configured in the "prefork" MPM. As soon as I configure it to the
"worker" mode (i.e. multi-threaded mode), the Apache process hangs whenever I
want so commit a file to one of my repositories, but this problem only occurs
as long as a "post-commit" hook is defined. Maybe other kinds of hooks are
affected as well, I haven't tried them, yet.
It seems for my that "mod_dav_svn" itself might be able to work in a
multi-threaded environment, but the connection to the hooks doesn't.
I've used the tool "truss" to capture the syscalls which are called by Apache
in both situations:
a) MPM Prefork
| 25769:
stat("/srv/www/vhosts/sirius/repo/modulhandbuch/svn/hooks/post-commit",
0xFFFF80F48B165B40) = 0
| 25769: open("/dev/null", O_WRONLY|O_CLOEXEC) = 24
| 25769: fcntl(24, F_DUPFD, 0x00000000) = 25
| 25769: fcntl(25, F_GETFD, 0x7FFA9304E144) = 0
| 25769: fcntl(25, F_SETFD, 0x00000000) = 0
| 25769: pipe() = 26 [27]
| 25769: fcntl(26, F_GETFD, 0x99427F850) = 0
| 25769: fcntl(26, F_SETFD, 0x00000001) = 0
| 25769: forkx(0) = 25782
| 25769: lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000, 0x00000000,
0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
| 25769: close(25) = 0
| 25769: close(27) = 0
| 25782: forkx() (returning as child ...) = 25769
| 25782: getpid() = 25782 [25769]
| 25782: munmap(0x7FFA91DC0000, 262144) = 0
| 25782: lwp_self() = 1
| 25782: lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000, 0x00000000,
0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
| 25782: close(23) = 0
| 25782: close(22) = 0
| 25782: close(21) = 0
| 25782: schedctl() = 0x7FFA92243000
| 25782: munmap(0x7FFA91A80000, 288862) = 0
| 25782: munmap(0x7FFA91AD7000, 7936) = 0
| 25782: munmap(0x7FFA91A60000, 29785) = 0
| 25782: munmap(0x7FFA91A78000, 2792) = 0
| 25782: close(5) = 0
| 25782: close(3) = 0
| 25782: close(6) = 0
| 25782: close(4) = 0
| 25782: close(14) = 0
| 25782: close(13) = 0
| 25782: close(12) = 0
| 25782: close(11) = 0
| 25782: close(10) = 0
| 25782: close(9) = 0
| 25782: close(8) = 0
| 25782: close(20) = 0
| 25782: close(26) = 0
| 25782: close(24) = 0
| 25782: fcntl(25, F_DUP2FD, 0x00000001) = 1
| 25782: close(25) = 0
| 25782: fcntl(27, F_DUP2FD, 0x00000002) = 2
| 25782: close(27) = 0
| 25782: sigaction(SIGCLD, 0xFFFF80F48B165A90, 0xFFFF80F48B165B20) = 0
| 25782: chdir(".") = 0
| 25782:
execve("/srv/www/vhosts/sirius/repo/modulhandbuch/svn/hooks/post-commit",
0x9942814C8, 0xFFFF80F48B165B78) argc = 4
| 25782: sysinfo(SI_MACHINE, "i86pc", 257) = 6
| 25782: mmap(0x00000000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,
4294967295, 0) = 0xFFFF80FFBF760000
| 25782: memcntl(0xFFFF80FFBF792000, 110504, MC_ADVISE, MADV_WILLNEED, 0, 0)
= 0
| 25782: memcntl(0x00400000, 172672, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
| 25782: resolvepath("/usr/bin/amd64/ksh93", "/usr/bin/amd64/ksh93", 1023) =
20
| 25782: resolvepath("/usr/lib/amd64/ld.so.1", "/lib/amd64/ld.so.1", 1023) =
18
| 25782: stat("/usr/bin/amd64/ksh93", 0xFFFF80FFBFFFFA50) = 0
| 25782: open("/var/ld/64/ld.config", O_RDONLY) = 3
| 25782: fstat(3, 0xFFFF80FFBFFFF7E0) = 0
| 25782: mmap(0x00000000, 156, PROT_READ, MAP_SHARED, 3, 0) =
0xFFFF80FFBF750000
b) MPM worker:
| 25747/27:
stat("/srv/www/vhosts/sirius/repo/modulhandbuch/svn/hooks/post-commit",
0x7FF86F7DC980) = 0
| 25747/27: open("/dev/null", O_WRONLY|O_CLOEXEC) = 24
| 25747/27: fcntl(24, F_DUPFD, 0x00000000) = 25
| 25747/27: fcntl(25, F_GETFD, 0x7FF87D69E144) = 0
| 25747/27: fcntl(25, F_SETFD, 0x00000000) = 0
| 25747/27: pipe() = 26 [27]
| 25747/27: fcntl(26, F_GETFD, 0xB075736D0) = 0
| 25747/27: fcntl(26, F_SETFD, 0x00000001) = 0
| 25747/27: lwp_suspend(28) = 0
| 25747/27: lwp_suspend(1) = 0
| 25747/27: lwp_suspend(3) = 0
| 25747/27: lwp_suspend(4) = 0
| 25747/27: lwp_suspend(5) = 0
| 25747/27: lwp_suspend(6) = 0
| 25747/27: lwp_suspend(7) = 0
| 25747/27: lwp_suspend(8) = 0
| 25747/27: lwp_suspend(9) = 0
| 25747/27: lwp_suspend(10) = 0
| 25747/27: lwp_suspend(11) = 0
| 25747/27: lwp_suspend(12) = 0
| 25747/27: lwp_suspend(13) = 0
| 25747/27: lwp_suspend(14) = 0
| 25747/27: lwp_suspend(15) = 0
| 25747/27: lwp_suspend(16) = 0
| 25747/27: lwp_suspend(17) = 0
| 25747/27: lwp_suspend(18) = 0
| 25747/27: lwp_suspend(19) = 0
| 25747/27: lwp_suspend(20) = 0
| 25747/27: lwp_suspend(21) = 0
| 25747/27: lwp_suspend(22) = 0
| 25747/27: lwp_suspend(23) = 0
| 25747/27: lwp_suspend(24) = 0
| 25747/27: lwp_suspend(25) = 0
| 25747/27: lwp_suspend(26) = 0
| 25747/27: forkx(0) = 25753
| 25747/27: lwp_continue(28) = 0
| 25747/27: lwp_continue(1) = 0
| 25747/27: lwp_continue(3) = 0
| 25753: forkx() (returning as child ...) = 25747
| 25747/27: lwp_continue(4) = 0
| 25747/27: lwp_continue(5) = 0
| 25747/27: lwp_continue(6) = 0
| 25747/27: lwp_continue(7) = 0
| 25753: getpid() = 25753 [25747]
| 25747/27: lwp_continue(8) = 0
| 25747/27: lwp_continue(9) = 0
| 25747/27: lwp_continue(10) = 0
| 25747/27: lwp_continue(11) = 0
| 25747/27: lwp_continue(12) = 0
| 25753: schedctl() = 0x7FF87C885000
| 25747/27: lwp_continue(13) = 0
| 25747/27: lwp_continue(14) = 0
| 25753: munmap(0x7FF87C0D0000, 262144) = 0
| 25747/27: lwp_continue(15) = 0
| 25747/27: lwp_continue(16) = 0
| 25747/27: lwp_continue(17) = 0
| 25747/27: lwp_continue(18) = 0
| 25753: lwp_self() = 27
| 25747/27: lwp_continue(19) = 0
| 25747/27: lwp_continue(20) = 0
| 25747/27: lwp_continue(21) = 0
| 25747/27: lwp_continue(22) = 0
| 25747/27: lwp_continue(23) = 0
| 25747/27: lwp_continue(24) = 0
| 25747/27: lwp_continue(25) = 0
| 25747/27: lwp_continue(26) = 0
| 25747/27: lwp_sigmask(SIG_SETMASK, 0xFFBE6007, 0xFFFFFFF7, 0x000000FF,
0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
| 25753: munmap(0x7FF87B800000, 8245248) = 0
| 25753: munmap(0x7FF86E800000, 8245248) = 0
| 25753: munmap(0x7FF87B000000, 8245248) = 0
| 25747/27: close(25) = 0
| 25747/27: close(27) = 0
| 25753: munmap(0x7FF87A800000, 8245248) = 0
| 25753: munmap(0x7FF87A000000, 8245248) = 0
| 25753: munmap(0x7FF879800000, 8245248) = 0
| 25753: munmap(0x7FF879000000, 8245248) = 0
| 25753: munmap(0x7FF878800000, 8245248) = 0
| 25753: munmap(0x7FF878000000, 8245248) = 0
| 25753: munmap(0x7FF877800000, 8245248) = 0
| 25753: munmap(0x7FF877000000, 8245248) = 0
| 25753: munmap(0x7FF876800000, 8245248) = 0
| 25753: munmap(0x7FF876000000, 8245248) = 0
| 25753: munmap(0x7FF875800000, 8245248) = 0
| 25753: munmap(0x7FF875000000, 8245248) = 0
| 25753: munmap(0x7FF874800000, 8245248) = 0
| 25753: munmap(0x7FF874000000, 8245248) = 0
| 25753: munmap(0x7FF873800000, 8245248) = 0
| 25753: munmap(0x7FF873000000, 8245248) = 0
| 25753: munmap(0x7FF872800000, 8245248) = 0
| 25753: munmap(0x7FF872000000, 8245248) = 0
| 25753: munmap(0x7FF871800000, 8245248) = 0
| 25753: munmap(0x7FF871000000, 8245248) = 0
| 25753: munmap(0x7FF870800000, 8245248) = 0
| 25753: munmap(0x7FF870000000, 8245248) = 0
| 25753: munmap(0x7FF86F800000, 8245248) = 0
| 25753: lwp_sigmask(SIG_SETMASK, 0xFFBE6007, 0xFFFFFFF7, 0x000000FF,
0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
| 25753: close(23) = 0
| 25753: close(22) = 0
| 25753: close(21) = 0
| 25747/28: lwp_mutex_timedlock(0x7FF87C180000, 0x00000000,
0x7FF87C19DA40) (sleeping...)
| 25749/28: port_getn(19, 0xB06513E50, 2, 1, 0x00000000) (sleeping...)
| 25743: pollsys(0xFFFF80F7BA722F48, 1, 0xFFFF80F7BA722F20, 0x00000000) = 0
| 25743: lwp_mutex_timedlock(0x7FF87C1B0000, 0x00000000, 0x7FF87D592A40) = 0
| 25743: lwp_mutex_unlock(0x7FF87C1B0000) = 0
| 25743: lwp_mutex_timedlock(0x7FF87C1B0000, 0x00000000, 0x7FF87D592A40) = 0
| 25743: lwp_mutex_unlock(0x7FF87C1B0000) = 0
| 25743: lwp_mutex_timedlock(0x7FF87C1B0000, 0x00000000, 0x7FF87D592A40) = 0
| 25743: lwp_mutex_unlock(0x7FF87C1B0000) = 0
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000)
(sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) = 0
| 25741: waitid(P_ALL, 0, 0xFFFF80F7BA726B50,
WEXITED|WTRAPPED|WSTOPPED|WNOHANG) = 0
| 25747/27: read(26, 0xB075CB9B8, 16384) (sleeping...)
| 25753: lwp_park(0x00000000, 0) (sleeping...)
| 25743: pollsys(0xFFFF80F7BA722F48, 1, 0xFFFF80F7BA722F20, 0x00000000)
(sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) = 0
| 25741: waitid(P_ALL, 0, 0xFFFF80F7BA726B50,
WEXITED|WTRAPPED|WSTOPPED|WNOHANG) = 0
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000)
(sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) = 0
| 25741: waitid(P_ALL, 0, 0xFFFF80F7BA726B50,
WEXITED|WTRAPPED|WSTOPPED|WNOHANG) = 0
| 25743: pollsys(0xFFFF80F7BA722F48, 1, 0xFFFF80F7BA722F20, 0x00000000) = 0
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000)
(sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) = 0
| 25741: waitid(P_ALL, 0, 0xFFFF80F7BA726B50,
WEXITED|WTRAPPED|WSTOPPED|WNOHANG) = 0
| 25743: pollsys(0xFFFF80F7BA722F48, 1, 0xFFFF80F7BA722F20, 0x00000000)
(sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000)
(sleeping...)
| 25741: pollsys(0xFFFF80F7BA726C10, 0, 0xFFFF80F7BA726C90, 0x00000000) = 0
| 25741: waitid(P_ALL, 0, 0xFFFF80F7BA726B50,
WEXITED|WTRAPPED|WSTOPPED|WNOHANG) = 0
As you can see, in "prefork" mode the hook is actually executed using the
syscall "execve()". All is running fine then. In the "worker" mode, the process
25753 is created by the "forkx(0)" call, bit it will go to the state
"lwp_park(0x00000000, 0) (sleeping...)" were it never awakes from anymore. I
actually have to kill the hanging process usind a SIGKILL. The SVN client which
committed to the repository via https will also hang forever. The file(s) I
want to commit are actually committed, i.e. the whole problem occurs after
commit - which is quite feasible because it is a "post-commit" hook. As the
"post-commit" script file is never executed, it doesn't matter what the
contents of the "post-commit" script are.
So my questions to you are:
1) Are there any known multi-threading problems with the combination Apache
2.4 MPM Worker, SVN 1.7.x, MOD_DAV_SVN and hooks?
2) Is there any alternative to access an SVN repository via Apache besides
using the MOD_DAV_SVN module in Apache? For example, can I possibly use "Fast
CGI" to link SVN to Apache? (This is, for example, a solution to run the
non-thread-safe PHP in a multi-threaded Worker MPM).
3) What is the recommended way to solve the problem? Do I really have to go
without a multi-threaded web server?
Any help or information to solve my problem is highly appreciated! Thank you
very much in advance!
Kind regards,
Steffen
--
------------------------------------------------------------------------
Dipl.-Inf. Steffen Moser
School of Advanced Professional Studies Room: O27/317
Ulm University Tel: +49.731.50-24179
Albert-Einstein-Allee 11 Fax: +49.731.50-24182
89081 Ulm, Germany http://saps.uni-ulm.de/
