When building ClamAV with -D_GLIBCXX_PARALLEL -fopenmp, clamd hangs, attaching
gdb to the hanged process reveals 2 threads, one is waiting in poll (normal),
the other one is hanged in gomp_team_barrier_wait_end.
Since there are no other threads that could grant it the barrier, that thread
will wait there indefinetely:
2 Thread 0x7fedad8d1910 (LWP 6147) 0x00000031798c0783 in *__GI___poll
(fds=<value optimized out>, nfds=<value optimized out>,
timeout=<value optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
* 1 Thread 0x7fedbdc84770 (LWP 6146) futex_wait (bar=<value optimized out>,
state=<value optimized out>)
at ../../../src/libgomp/config/linux/x86/futex.h:44
Steps to reproduce:
1. Download ClamAV snapshot:
http://git.clamav.net/gitweb?p=clamav-devel.git;a=snapshot;h=fc382bd68b9e2e14198ca05efc72fba15f1a32da;sf=tgz
2. Unpack snapshot
3. Build it with libstdc++ parallel mode:
$ ./configure CXXFLAGS=-fopenmp -D_GLIBCXX_PARALLEL LDFLAGS=-fopenmp
--disable-clamav --enable-llvm=yes
$ make -j4
4. Edit etc/clamd.conf, remove "Example" line
5. Start clamd:
$ clamd/clamd -c etc/clamd.conf
6. Reload database twice:
$ clamdscan/clamdscan --reload -c etc/clamd.conf
$ clamdscan/clamdscan --reload -c etc/clamd.conf
^C(deadlocks here)
7. $ ps -ef | grep clamd
edwin 7226 1 0 19:55 ? 00:00:00
/home/edwin/clam/git/publicgitgit/clamd/.libs/lt-clamd -c etc/clamd.conf
8. Attach gdb
$ gdb
(gdb) attach 7226
(gdb) thread apply all bt
Thread 2 (Thread 0x7f5180e31910 (LWP 7294)):
#0 0x00000031798c0783 in *__GI___poll (fds=<value optimized out>, nfds=<value
optimized out>, timeout=<value optimized out>)
at ../sysdeps/unix/sysv/linux/poll.c:87
#1 0x000000000040ee6d in fds_poll_recv (data=0x7fff884c1e40, timeout=<value
optimized out>, check_signals=0) at others.c:487
#2 0x000000000040d31f in acceptloop_th (arg=<value optimized out>) at
server-th.c:320
#3 0x000000317a40673a in start_thread (arg=<value optimized out>) at
pthread_create.c:300
#4 0x00000031798cb6fd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f51911e4770 (LWP 7293)):
#0 futex_wait (bar=<value optimized out>, state=<value optimized out>) at
../../../src/libgomp/config/linux/x86/futex.h:44
#1 do_wait (bar=<value optimized out>, state=<value optimized out>) at
../../../src/libgomp/config/linux/wait.h:58
#2 gomp_team_barrier_wait_end (bar=<value optimized out>, state=<value
optimized out>)
at ../../../src/libgomp/config/linux/bar.c:109
#3 0x00007f51913d43d8 in
__gnu_parallel::find_template<__gnu_cxx::__normal_iterator<llvm::PassInfo
const* const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >,
__gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, std::binder2nd<__gnu_parallel::equal_to<llvm::PassInfo const*,
llvm::PassInfo const* const&> >, __gnu_parallel::find_if_selector> ()
from /home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#4 0x00007f51913defea in std::pair<__gnu_cxx::__normal_iterator<llvm::PassInfo
const* const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >,
__gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > > >
__gnu_parallel::find_template<__gnu_cxx::__normal_iterator<llvm::PassInfo
const* const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >,
__gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, std::binder2nd<__gnu_parallel::equal_to<llvm::PassInfo const*,
llvm::PassInfo const* const&> >,
__gnu_parallel::find_if_selector>(__gnu_cxx::__normal_iterator<llvm::PassInfo
const* const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >,
__gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, __gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, std::binder2nd<__gnu_parallel::equal_to<llvm::PassInfo const*,
llvm::PassInfo const* const&> >, __gnu_parallel::find_if_selector,
__gnu_parallel::constant_size_blocks_tag) ()
from /home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#5 0x00007f51913dc9b1 in std::pair<__gnu_cxx::__normal_iterator<llvm::PassInfo
const* const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >,
__gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > > >
__gnu_parallel::find_template<__gnu_cxx::__normal_iterator<llvm::PassInfo
const* const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >,
__gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, std::binder2nd<__gnu_parallel::equal_to<llvm::PassInfo const*,
llvm::PassInfo const* const&> >,
__gnu_parallel::find_if_selector>(__gnu_cxx::__normal_iterator<llvm::PassInfo
const* const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >,
__gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, __gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, std::binder2nd<__gnu_parallel::equal_to<llvm::PassInfo const*,
llvm::PassInfo const* const&> >, __gnu_parallel::find_if_selector) ()
from /home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#6 0x00007f51913daaff in __gnu_cxx::__normal_iterator<llvm::PassInfo const*
const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >
std::__parallel::find_switch<__gnu_cxx::__normal_iterator<llvm::PassInfo const*
const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >, llvm::PassInfo
const*>(__gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, __gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, llvm::PassInfo const* const&, std::random_access_iterator_tag) ()
from /home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#7 0x00007f51913d7f6c in __gnu_cxx::__normal_iterator<llvm::PassInfo const*
const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >
std::__parallel::find<__gnu_cxx::__normal_iterator<llvm::PassInfo const*
const*, std::__cxx1998::vector<llvm::PassInfo const*,
std::allocator<llvm::PassInfo const*> > >, llvm::PassInfo
const*>(__gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, __gnu_cxx::__normal_iterator<llvm::PassInfo const* const*,
std::__cxx1998::vector<llvm::PassInfo const*, std::allocator<llvm::PassInfo
const*> > >, llvm::PassInfo const* const&) () from
/home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#8 0x00007f51913ce5fa in
llvm::PMTopLevelManager::findAnalysisPass(llvm::PassInfo const*) ()
from /home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#9 0x00007f51913ce2fb in llvm::PMTopLevelManager::schedulePass(llvm::Pass*) ()
from /home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#10 0x00007f51913d5692 in llvm::FunctionPassManagerImpl::add(llvm::Pass*) ()
from /home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#11 0x00007f51913d18df in llvm::FunctionPassManager::add(llvm::Pass*) ()
from /home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#12 0x00007f5191906e47 in
llvm::LLVMTargetMachine::addCommonCodeGenPasses(llvm::PassManagerBase&,
llvm::CodeGenOpt::Level) ()
from /home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#13 0x00007f5191906c01 in
llvm::LLVMTargetMachine::addPassesToEmitMachineCode(llvm::PassManagerBase&,
llvm::JITCodeEmitter&, llvm::CodeGenOpt::Level) () from
/home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#14 0x00007f519139165e in llvm::JIT::JIT(llvm::ModuleProvider*,
llvm::TargetMachine&, llvm::TargetJITInfo&, llvm::JITMemoryManager*,
llvm::CodeGenOpt::Level, bool) () from
/home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#15 0x00007f5191391209 in llvm::JIT::createJIT(llvm::ModuleProvider*,
std::string*, llvm::JITMemoryManager*, llvm::CodeGenOpt::Level, bool,
llvm::CodeModel::Model) () from
/home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#16 0x00007f5191391131 in
llvm::ExecutionEngine::createJIT(llvm::ModuleProvider*, std::string*,
llvm::JITMemoryManager*, llvm::CodeGenOpt::Level, bool, llvm::CodeModel::Model)
() from /home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#17 0x00007f51913948ec in llvm::JIT::create(llvm::ModuleProvider*,
std::string*, llvm::JITMemoryManager*, llvm::CodeGenOpt::Level, bool,
llvm::CodeModel::Model) () from
/home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#18 0x00007f5191544b3f in llvm::EngineBuilder::create() () from
/home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#19 0x00007f519132d365 in cli_bytecode_prepare_jit () from
/home/edwin/clam/git/publicgitgit/libclamav/.libs/libclamav.so.6
#20 0x00007f5191310de8 in cli_bytecode_prepare (bcs=0x1703104) at
bytecode.c:1580
#21 0x00007f51912a380d in cl_engine_compile (engine=0x17009f0) at readdb.c:2614
#22 0x000000000040b0ea in reload_db (engine=0x17009f0, dboptions=8234,
opts=<value optimized out>,
do_check=<value optimized out>, ret=0x7fff884c216c) at server-th.c:231
#23 0x000000000040c426 in recvloop_th (socketds=0x0, nsockets=<value optimized
out>, engine=<value optimized out>,
dboptions=<value optimized out>, opts=<value optimized out>) at
server-th.c:1265
#24 0x0000000000407b4b in main (argc=<value optimized out>, argv=<value
optimized out>) at clamd.c:486
(gdb)
9. See that in the thread backtrace one thread is waiting in poll(), other one
in gomp_team_barrier_end, and those are only 2 threads.
10. Deadlock
System info:
$ g++ -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.4.2-8'
--with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared
--enable-multiarch --enable-linker-build-id --with-system-zlib
--libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
--with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls
--enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc
--with-arch-32=i486 --with-tune=generic --enable-checking=release
--build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.4.2 (Debian 4.4.2-8)
Sorry that I didn't reduce a testcase, a trivial testcase with std::find and
pthread doesn't hang.
There is also a problem with parallel mode when building LLVM, LLVM's tblgen
makes very slow progress when multiple tblgen processes are executing in
parallel on a multicore machine (see http://llvm.org/bugs/show_bug.cgi?id=5804)
--
Summary: libstdc++ parallel mode deadlocks in barrier
Product: gcc
Version: unknown
Status: UNCONFIRMED
Severity: normal
Priority: P3
Component: libstdc++
AssignedTo: unassigned at gcc dot gnu dot org
ReportedBy: edwintorok at gmail dot com
GCC build triplet: x86_64-linux-gnu
GCC host triplet: x86_64-linux-gnu
GCC target triplet: x86_64-linux-gnu
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=42624
