Hi!
After a recent upgrade to svn 1.8.5, we are facing a problem with
threaded svnserve: after some period of working without problem it hangs
and eats 100% cpu.
I've captured stack traces from all svnserve threads, and it looks like
the problem is in libsvn_subr/cache-membuffer.c, function
ensure_data_insertable() - one thread is doing an infinite loop inside
it with write lock acquired on the cache, and other threads are waiting
for that one to release his lock.
There was no problem in 1.6 or 1.7; by no problem I mean not only the
absence of hang, but also the absence of 100% cpu consumption => absence
of infinite loop.
So it's likely that the new part of cache eviction logic which is
executed if (cache->hit_count <= cache->used_entries) fails to terminate
the loop sometimes, because all other code from that function was
already there in 1.7.
Sorry for not providing a core dump, I didn't immediately think of
capturing it, but the error happens almost every day, so the next time
I'll try to capture and dig it deeper.
Stack traces are attached.
--
With best regards,
Vitaliy Filippov
Stack trace from 'running' (while(1)) thread:
#0 0xb7646831 in move_entry (cache=cache@entry=0xb4ff9058, entry=0xb4f86a1c)
at /root/svn/subversion-1.8.5/subversion/libsvn_subr/cache-membuffer.c:951
#1 0xb7646aa1 in ensure_data_insertable (cache=cache@entry=0xb4ff9058,
size=size@entry=3378689)
at /root/svn/subversion-1.8.5/subversion/libsvn_subr/cache-membuffer.c:1090
#2 0xb7647a42 in membuffer_cache_set_internal (size=3378689, buffer=0xb5eea028
"\320\317\021\340\241\261", <incomplete sequence \341>,
group_index=<error reading variable: Could not find type for
DW_OP_GNU_const_type>, to_find=0xb691d97c, cache=<optimized out>,
scratch_pool=<optimized out>)
at /root/svn/subversion-1.8.5/subversion/libsvn_subr/cache-membuffer.c:1400
#3 membuffer_cache_set (scratch_pool=<optimized out>, serializer=0x338e01,
item=0xb5eea018, key=0xb691d97c, cache=<optimized out>)
at /root/svn/subversion-1.8.5/subversion/libsvn_subr/cache-membuffer.c:1476
#4 svn_membuffer_cache_set (cache_void=0xb691d958, key=0xb6877e64,
value=0xb5eea018, scratch_pool=0xb6892018)
at /root/svn/subversion-1.8.5/subversion/libsvn_subr/cache-membuffer.c:2012
#5 0xb7648ee4 in svn_cache__set (cache=0xb691d928, key=key@entry=0xb6877e64,
value=0xb5eea018, scratch_pool=0xb6892018)
at /root/svn/subversion-1.8.5/subversion/libsvn_subr/cache.c:105
#6 0xb73c9332 in rep_read_contents (len=0xb21fdbec,
buf=0xb655e018
"\351\373\345\366\373\374v\256\302\203\241\363\334P3\365\227\255\373QW`\367t\377D\370P9BV\377\327KThƢU\b`cl\375\200\252\242nw\213\377BK\271_⢳\241?H\\~\362H\363Qe~\022\033\261DI^\377\177BG\\\377CH]Nb\034\347\241\v\261ej\177\252\022Ka\247\361\347OTif\022\033\261KQd\376z\020F\377\021\027$\377~\377~\214\377\320\312\333\377\271\377\263\306\377\273\267\312\377\270\357\267\313\377\251\214Щ\251\301\377\377\242\242\272\377\241\235\266\377\377\243\237\270\377\251\246\300\377\377\233\234\266\377\255\267\317\377\377r\203\236\377o\213\252\377\377^~\241\377Rp\223\377\377`|\237\377"...,
baton=0xb6877e28) at
/root/svn/subversion-1.8.5/subversion/libsvn_fs_fs/fs_fs.c:5256
#7 rep_read_contents (baton=0xb6877e28,
buf=0xb655e018
"\351\373\345\366\373\374v\256\302\203\241\363\334P3\365\227\255\373QW`\367t\377D\370P9BV\377\327KThƢU\b`cl\375\200\252\242nw\213\377BK\271_⢳\241?H\\~\362H\363Qe~\022\033\261DI^\377\177BG\\\377CH]Nb\034\347\241\v\261ej\177\252\022Ka\247\361\347OTif\022\033\261KQd\376z\020F\377\021\027$\377~\377~\214\377\320\312\333\377\271\377\263\306\377\273\267\312\377\270\357\267\313\377\251\214Щ\251\301\377\377\242\242\272\377\241\235\266\377\377\243\237\270\377\251\246\300\377\377\233\234\266\377\255\267\317\377\377r\203\236\377o\213\252\377\377^~\241\377Rp\223\377\377`|\237\377"...,
len=0xb21fdbec) at
/root/svn/subversion-1.8.5/subversion/libsvn_fs_fs/fs_fs.c:5217
#8 0xb7678e48 in svn_stream_read (stream=0xb6877f00,
buffer=0xb655e018
"\351\373\345\366\373\374v\256\302\203\241\363\334P3\365\227\255\373QW`\367t\377D\370P9BV\377\327KThƢU\b`cl\375\200\252\242nw\213\377BK\271_⢳\241?H\\~\362H\363Qe~\022\033\261DI^\377\177BG\\\377CH]Nb\034\347\241\v\261ej\177\252\022Ka\247\361\347OTif\022\033\261KQd\376z\020F\377\021\027$\377~\377~\214\377\320\312\333\377\271\377\263\306\377\273\267\312\377\270\357\267\313\377\251\214Щ\251\301\377\377\242\242\272\377\241\235\266\377\377\243\237\270\377\251\246\300\377\377\233\234\266\377\255\267\317\377\377r\203\236\377o\213\252\377\377^~\241\377Rp\223\377\377`|\237\377"...,
len=len@entry=0xb21fdbec)
at /root/svn/subversion-1.8.5/subversion/libsvn_subr/stream.c:144
#9 0xb76b63fd in txdelta_next_window (window=0xb21fdc4c, baton=0xb6877f28,
pool=0xb6820018) at
/root/svn/subversion-1.8.5/subversion/libsvn_delta/text_delta.c:376
#10 0xb76b666f in svn_txdelta_next_window (window=window@entry=0xb21fdc4c,
stream=stream@entry=0xb6590018, pool=pool@entry=0xb6820018)
at /root/svn/subversion-1.8.5/subversion/libsvn_delta/text_delta.c:346
#11 0xb76b758c in svn_txdelta_send_txstream (txstream=0xb6590018,
handler=0xb76b4fd0 <window_handler>, handler_baton=0xb68ad058,
pool=pool@entry=0xb6876018)
at /root/svn/subversion-1.8.5/subversion/libsvn_delta/text_delta.c:992
#12 0xb76e9de7 in delta_files (b=b@entry=0xb68e4428, file_baton=<optimized
out>, s_rev=28624, s_path=0x0,
t_path=0xb6876098 "/rms/doc/Разделение RMS/ÐеÑодÑ
ÑазделениÑ.vsd", lock_token=0x0, pool=0xb6876018)
at /root/svn/subversion-1.8.5/subversion/libsvn_repos/reporter.c:736
#13 0xb76eaac0 in update_entry (b=b@entry=0xb68e4428, s_rev=s_rev@entry=28624,
s_path=s_path@entry=0x0, s_entry=s_entry@entry=0x0,
t_path=0xb6876098 "/rms/doc/Разделение RMS/ÐеÑодÑ
ÑазделениÑ.vsd", t_entry=t_entry@entry=0xb68b04a0,
dir_baton=dir_baton@entry=0xb68af1f0,
e_path=e_path@entry=0xb6876058 "Разделение RMS/ÐеÑодÑ
ÑазделениÑ.vsd", info=info@entry=0x0, wc_depth=svn_depth_infinity,
requested_depth=svn_depth_infinity, pool=pool@entry=0xb6876018) at
/root/svn/subversion-1.8.5/subversion/libsvn_repos/reporter.c:1066
#14 0xb76eb8b1 in delta_dirs (b=b@entry=0xb68e4428, s_rev=s_rev@entry=28624,
s_path=s_path@entry=0x0, t_path=t_path@entry=0xb68af078 "/rms/doc/Ð
азделение RMS",
dir_baton=0xb68af1f0, e_path=e_path@entry=0xb68af058 "Разделение
RMS", start_empty=0, wc_depth=wc_depth@entry=svn_depth_infinity,
requested_depth=requested_depth@entry=svn_depth_infinity,
pool=pool@entry=0xb68af018) at
/root/svn/subversion-1.8.5/subversion/libsvn_repos/reporter.c:1317
#15 0xb76eac09 in update_entry (b=b@entry=0xb68e4428, s_rev=s_rev@entry=28624,
s_path=s_path@entry=0x0, s_entry=s_entry@entry=0x0,
t_path=0xb68af078 "/rms/doc/Разделение RMS",
t_entry=t_entry@entry=0xb68b1d08, dir_baton=dir_baton@entry=0xb68d7688,
e_path=e_path@entry=0xb68af058 "Разделение RMS",
info=info@entry=0x0, wc_depth=svn_depth_infinity,
requested_depth=svn_depth_infinity,
pool=pool@entry=0xb68af018) at
/root/svn/subversion-1.8.5/subversion/libsvn_repos/reporter.c:1038
#16 0xb76eb8b1 in delta_dirs (b=b@entry=0xb68e4428, s_rev=s_rev@entry=28624,
s_path=s_path@entry=0xb68d71e8 "/rms/doc", t_path=0xb68e44b8 "/rms/doc",
dir_baton=0xb68d7688, e_path=e_path@entry=0xb76f26e1 "", start_empty=1,
wc_depth=svn_depth_infinity, requested_depth=svn_depth_infinity,
pool=pool@entry=0xb68d7018)
at /root/svn/subversion-1.8.5/subversion/libsvn_repos/reporter.c:1317
#17 0xb76ebf8a in drive (pool=0xb68d7018, info=0xb68d7120, s_rev=28624,
b=0xb68e4428) at
/root/svn/subversion-1.8.5/subversion/libsvn_repos/reporter.c:1381
#18 finish_report (pool=0xb68d7018, b=0xb68e4428) at
/root/svn/subversion-1.8.5/subversion/libsvn_repos/reporter.c:1445
#19 svn_repos_finish_report (baton=0xb68e4428, pool=0xb68d7018) at
/root/svn/subversion-1.8.5/subversion/libsvn_repos/reporter.c:1542
#20 0x080524d6 in ?? ()
#21 0xb76254fc in svn_ra_svn__handle_commands2 () from
/usr/lib/i386-linux-gnu/libsvn_ra_svn-1.so.1
#22 0x0804fbca in ?? ()
#23 0x08054f05 in ?? ()
#24 0xb76254fc in svn_ra_svn__handle_commands2 () from
/usr/lib/i386-linux-gnu/libsvn_ra_svn-1.so.1
#25 0x08057e48 in ?? ()
#26 0x08058155 in ?? ()
#27 0xb75ffff6 in ?? () from /usr/lib/i386-linux-gnu/libapr-1.so.0
#28 0xb75a8cf1 in start_thread () from
/lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#29 0xb74e1c3e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
Stack trace from all other, blocked threads:
#0 0xb771d424 in __kernel_vsyscall ()
#1 0xb75abe48 in pthread_rwlock_rdlock () from
/lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#2 0xb75f361e in apr_thread_rwlock_rdlock () from
/usr/lib/i386-linux-gnu/libapr-1.so.0
#3 0xb7646e22 in read_lock_cache (cache=<error reading variable: Unhandled
dwarf expression opcode 0xfa>)
at /root/svn/subversion-1.8.5/subversion/libsvn_subr/cache-membuffer.c:494
#4 0xb764715f in membuffer_cache_get (result_pool=0xb6693018,
deserializer=0xb73ddd70 <svn_fs_fs__deserialize_id>, item=0xafbff0ec,
key=0xb686b27c, cache=0xb4ff9058)
at /root/svn/subversion-1.8.5/subversion/libsvn_subr/cache-membuffer.c:1572
#5 svn_membuffer_cache_get (value_p=0xafbff0ec, found=0xafbff098,
cache_void=0xb686b258, key=0xafbff0d8, result_pool=0xb6693018)
at /root/svn/subversion-1.8.5/subversion/libsvn_subr/cache-membuffer.c:1966
#6 0xb7648e6b in svn_cache__get (value_p=value_p@entry=0xafbff0ec,
found=found@entry=0xafbff098, cache=0xb686b228, key=key@entry=0xafbff0d8,
result_pool=result_pool@entry=0xb6693018) at
/root/svn/subversion-1.8.5/subversion/libsvn_subr/cache.c:84
#7 0xb73d319b in svn_fs_fs__rev_get_root
(root_id_p=root_id_p@entry=0xafbff0ec, fs=fs@entry=0xb6752020, rev=rev@entry=0,
pool=pool@entry=0xb6693018)
at /root/svn/subversion-1.8.5/subversion/libsvn_fs_fs/fs_fs.c:3091
#8 0xb73c2d2e in svn_fs_fs__dag_revision_root (node_p=node_p@entry=0xafbff12c,
fs=fs@entry=0xb6752020, rev=rev@entry=0, pool=pool@entry=0xb6693018)
at /root/svn/subversion-1.8.5/subversion/libsvn_fs_fs/dag.c:611
#9 0xb73e0c90 in svn_fs_fs__revision_root (pool=0xb6693018, rev=0,
fs=0xb6752020, root_p=0xafbff1a8) at
/root/svn/subversion-1.8.5/subversion/libsvn_fs_fs/tree.c:631
#10 svn_fs_fs__revision_root (root_p=0xafbff1a8, fs=0xb6752020, rev=0,
pool=0xb6693018) at
/root/svn/subversion-1.8.5/subversion/libsvn_fs_fs/tree.c:622
#11 0xb76c3fcb in svn_fs_revision_root () from
/usr/lib/i386-linux-gnu/libsvn_fs-1.so.1
#12 0xb76edfb1 in svn_repos_has_capability (repos=0xb6942470, has=0xafbff258,
capability=<optimized out>, pool=0xb6942018)
at /root/svn/subversion-1.8.5/subversion/libsvn_repos/repos.c:1643
#13 0x08057d43 in ?? ()
#14 0x08058155 in ?? ()
#15 0xb75ffff6 in ?? () from /usr/lib/i386-linux-gnu/libapr-1.so.0
#16 0xb75a8cf1 in start_thread () from
/lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#17 0xb74e1c3e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
