On 17/12/14 09:25, Charlie Smurthwaite wrote:
Hi,
I am running an SVN HTTP server using Apache/2.4.10 with SVN 1.8.10
(r1615264) and I am often seeing very high kernel CPU load.
The CPU time seems to be consumed in the kernel by "_raw_spin_lock",
with the httpd processes spending much of their time waiting on calls
to "futex".
Here's a syscall analysis from svn:
http://paste.codebasehq.com/pastes/7qzt68lx2eghz2gjns
Here's the kernel CPU time analysis: http://i.imgur.com/37Ryt5V.png
Here's a snapshot from htop showing why this is a problem:
http://i.imgur.com/I3mDDbi.png
I'd appreciate if anyone could tell me whether I have likely
configured something incorrectly, whether there is an obvious
workaround, or whether this needs to be escalated as a bug, and if so,
to whom?
Thank you!
Charlie
My httpd is configured as follows:
root@storage02:~# /opt/subversion-server/bin/httpd -V
Server version: Apache/2.4.10 (Unix)
Server built: Nov 7 2014 15:16:58
Server's Module Magic Number: 20120211:36
Server loaded: APR 1.5.1, APR-UTIL 1.5.4
Compiled using: APR 1.5.1, APR-UTIL 1.5.4
Architecture: 64-bit
Server MPM: worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/opt/subversion-server"
-D SUEXEC_BIN="/opt/subversion-server/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
It was suggested that this might be related to memory management, so I
have looked into a couple of things:
1) I disabled swap on the server in question. The host had 25GB of RAM
free and should not have been swapping active memory, however I believe
disabling swap has solved the problem.
2) The server is running NUMA, rather than SMP memory configuration. I
suspect that this is the reason for the problem (though I have no
evidence) and that it is not specific to SVN/Apache.
Thanks to Bert on Freenode for his assistance in finding this
workaround. I suspect that changing NUMA config, or changing Apache's
threading model would also prevent the problem.
Charlie
