Hi Andreas.
On 29/05/2014 5:10 PM, Andreas Cadhalpun wrote:
Thanks for your help in finding the cause.
Since the logs are inconclusive, it would really help if you could get us a
backtrace, when clamd is hanging.
For this install clamav-dbg and gdb.
When it hangs again, please determine the PIDs of all clamd threads, e.g. with:
ps -eL | grep clamd
The second column contains this number.
For every thread of clamd, run (replacing <PID> with the actual number):
sudo gdb /usr/sbin/clamd <PID>
And then in gdb:
bt full
The output of this should help to figure out, where exactly clamd is hanging.
Okay, I have a hang, and I have the gdb output that you want.
First the PID and Threads:
# ps -eL | grep -P 'LWP|clamd'
PID LWP TTY TIME CMD
26124 26124 ? 00:00:11 clamd
26124 26126 ? 00:00:00 clamd
Then connecting to the first thread via gdb:
# gdb /usr/sbin/clamd 26124
GNU gdb (GDB) 7.6.2 (Debian 7.6.2-1.1+b1)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/clamd...Reading symbols from
/usr/lib/debug/usr/sbin/clamd...done.
done.
Attaching to program: /usr/sbin/clamd, process 26124
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Reading symbols from /usr/lib/libclamav.so.6...Reading symbols from
/usr/lib/debug/usr/lib/libclamav.so.6.1.23...done.
done.
Loaded symbols for /usr/lib/libclamav.so.6
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libpthread-2.18.so...done.
done.
[New LWP 26126]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libc-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libxml2.so.2
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /lib/x86_64-linux-gnu/libbz2.so.1.0...(no debugging
symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libbz2.so.1.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libltdl.so.7...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libltdl.so.7
Reading symbols from /usr/lib/x86_64-linux-gnu/libffi.so.6...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libffi.so.6
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libm-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libdl-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging
symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/ld-2.18.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols
found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/liblzma.so.5
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libnss_compat-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libnsl-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libnss_nis-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libnss_files-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
__lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or
directory.
Then running the bt full command:
(gdb) bt full
#0 __lll_lock_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1 0x00007ff480483467 in _L_lock_913 () from
/lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x00007ff480483290 in __GI___pthread_mutex_lock (mutex=0x6256a0
<logg_mutex>) at ../nptl/pthread_mutex_lock.c:79
__PRETTY_FUNCTION__ = "__pthread_mutex_lock"
type = 4294966784
#3 0x0000000000405f5f in logg (str=str@entry=0x41f8c9 "Reading databases from
%s\n") at ../shared/output.c:350
args = {{gp_offset = 16, fp_offset = 48, overflow_arg_area =
0x7ffff5a3e220, reg_save_area = 0x7ffff5a3e130}}
buffer = "Reading databases from
/var/lib/clamav\n\000natures)\n\000\000\000\020\000\000\000\340\037G\200\364\177\000\000\000\000\000\000\000\000\000\000\t\310\002\336\004\r\te\001",
'\000' <repeats 15 times>,
"h\233n\356\376Wag\205\376\234\355s\337\343\225|\000\000\000\000\000\000\000@\315\r\001\000\000\000\000\002\000\000\000\000\000\000\000\r\177H\200\364\177\000\000\000\000\000\000\000\000\000\000\320ޣ\365\377\177\000\000\n\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\340\340\243\365\377\177\000\000\v\000\000\000\000\000\000\000\366\005B\000\000\000\000\000M"...
abuffer = 0x0
buff = 0x7ffff5a3dd20 "Reading databases from /var/lib/clamav\n"
currtime = 1401850421
len = 1025
old_umask = <optimized out>
fl = {l_type = 2306, l_whence = 0, l_start = 41943360, l_len = 1, l_pid
= 33184}
#4 0x000000000040da12 in reload_db (ret=<synthetic pointer>, do_check=0, opts=0x109b280, dboptions=8202, engine=0x10b2eb0) at
server-th.c:200
dbdir = 0x109bcf0 "/var/lib/clamav"
retval = <optimized out>
sigs = 0
settings = 0x111dcf0
#5 recvloop_th (socketds=0x10b3b40, nsockets=1, engine=0x10b2eb0,
dboptions=dboptions@entry=8202, opts=0x109b280) at server-th.c:1399
new_sd = 1
max_threads = <optimized out>
max_queue = 100
readtimeout = <optimized out>
ret = 0
options = 4219447
timestr = "8\aT\201\364\177\000\000\027\347\361\240", '\000' <repeats 12 times>,
"\005\000\000\000\000\000\000"
sigact = {__sigaction_handler = {sa_handler = 0x40c7b0 <sighandler_th>, sa_sigaction = 0x40c7b0 <sighandler_th>}, sa_mask =
{__val = {22531,
0 <repeats 15 times>}}, sa_flags = 0, sa_restorer = 0x0}
sigset = {__val = {18446744067266421556, 18446744073709551615 <repeats 15
times>}}
rlim = {rlim_cur = 1024, rlim_max = 4096}
old_umask = <optimized out>
opt = <optimized out>
buff =
"\000\347\361\240\000\000\000\000`\350\243\365\377\177\000\000\377\377\377\377\000\000\000\000\374\333\063\201\364\177\000\000\340\215\274e\364\177\000\000)\333\063\201\364\177\000\000\003\000\000\000\000\000\000\000\374\333\063\201\364\177\000\000\002\000\000\000\000\000\000\000!\317\t\375\000\000\000\000\003\000\000\000\000\000\000\000\021\000\000\000\000\000\000\000\060HT\201\364\177\000\000n\345\063\201\364\177\000\000\020\237G\200\364\177\000\000\220\346\243\365\377\177\000\000(-\r\200\364\177\000\000`\227\r\200\364\177\000\000\240\347\243\365\377\177\000\000<'\364\003\000\000\000\000\220\347\243\365\377\177",
'\000' <repeats 18 times>, "\250j\017\001\000\000\000\000\000\200T\201\364\177\000\000"...
mainpid = <optimized out>
idletimeout = <optimized out>
val = <optimized out>
i = <optimized out>
j = <optimized out>
rr_last = 1
accept_th = 140687645054720
fds_mutex = {__data = {__lock = 1, __count = 0, __owner = 26126, __nusers = 1, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0,
__next = 0x0}}, __size =
"\001\000\000\000\000\000\000\000\016f\000\000\001", '\000' <repeats 26 times>,
__align = 1}
recvfds_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0,
__list = {__prev = 0x0,
__next = 0x0}}, __size = '\000' <repeats 12 times>, "\001", '\000'
<repeats 26 times>, __align = 0}
acceptdata = {fds = {buf_mutex = 0x7ffff5a3e4a0, buf = 0x10b20a0, nfds = 2, poll_data = 0x7ff4600008c0, poll_data_nfds =
2}, recv_fds = {
buf_mutex = 0x7ffff5a3e4d0, buf = 0x10dd250, nfds = 101, poll_data = 0x10b3aa0, poll_data_nfds = 2}, cond_nfds =
{__data = {__lock = 0,
__futex = 1, __total_seq = 1, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x7ffff5a3e4d0, __nwaiters = 2,
__broadcast_seq = 0},
__size = "\000\000\000\000\001\000\000\000\001", '\000' <repeats 23 times>,
"\320\344\243\365\377\177\000\000\002\000\000\000\000\000\000",
__align = 4294967296}, max_queue = 100, commandtimeout = 5,
syncpipe_wake_recv = {5, 6}, syncpipe_wake_accept = {7, 8}}
---Type <return> to continue, or q <return> to quit---
fds = 0x7ffff5a3e568
start_time = 1401850421
current_time = 1401850421
selfchk = 3600
thr_pool = 0x10fc390
fan_pid = 140687653447424
fan_attr = {__size = '\000' <repeats 17 times>, "\020", '\000' <repeats 37
times>, __align = 0}
tharg = 0x10d1e50
#6 0x0000000000405064 in main (argc=<optimized out>, argv=<optimized out>) at
clamd.c:721
engine = 0x109fe70
opt = <optimized out>
user = <optimized out>
sa = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags =
0, sa_restorer = 0x0}
rlim = {rlim_cur = 140688097094952, rlim_max = 140688097145256}
currtime = 1401834875
dbdir = 0x109bcf0 "/var/lib/clamav"
cfgfile = <optimized out>
pua_cats = <optimized out>
pt = <optimized out>
ret = <optimized out>
tcpsock = <optimized out>
localsock = <optimized out>
i = <optimized out>
min_port = <optimized out>
max_port = <optimized out>
sigs = 4602067
lsockets = 0x10b3b40
nlsockets = 1
dboptions = 8202
sb = {st_dev = 2, st_ino = 1, st_nlink = 290, st_mode = 16749, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size =
0, st_blksize = 1024,
st_blocks = 0, st_atim = {tv_sec = 1399574165, tv_nsec = 20000000}, st_mtim = {tv_sec = 1399574165, tv_nsec = 20000000},
st_ctim = {
tv_sec = 1399574165, tv_nsec = 20000000}, __unused = {0, 0, 0}}
Then I quit gdb:
(gdb) quit
A debugging session is active.
Inferior 1 [process 26124] will be detached.
Quit anyway? (y or n) y
Detaching from program: /usr/sbin/clamd, process 26124
Then I attach gdb to the other thread:
# gdb /usr/sbin/clamd 26126
GNU gdb (GDB) 7.6.2 (Debian 7.6.2-1.1+b1)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/clamd...Reading symbols from
/usr/lib/debug/usr/sbin/clamd...done.
done.
Attaching to program: /usr/sbin/clamd, process 26126
warning: process 26126 is a cloned process
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Reading symbols from /usr/lib/libclamav.so.6...Reading symbols from
/usr/lib/debug/usr/lib/libclamav.so.6.1.23...done.
done.
Loaded symbols for /usr/lib/libclamav.so.6
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libpthread-2.18.so...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libc-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libxml2.so.2
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /lib/x86_64-linux-gnu/libbz2.so.1.0...(no debugging
symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libbz2.so.1.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libltdl.so.7...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libltdl.so.7
Reading symbols from /usr/lib/x86_64-linux-gnu/libffi.so.6...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libffi.so.6
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libm-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libdl-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging
symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/ld-2.18.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols
found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/liblzma.so.5
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libnss_compat-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libnsl-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libnss_nis-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libnss_files-2.18.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
warning: no loadable sections found in added symbol-file system-supplied DSO at
0x7ffff5b4d000
pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such
file or directory.
Then run the gdb bt full command:
(gdb) bt full
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1 0x000000000040c421 in acceptloop_th (arg=0x7ffff5a3e540) at server-th.c:378
buf = 0x10b20a0
new_sd = <optimized out>
buff = '\000' <repeats 1024 times>
i = 0
data = 0x7ffff5a3e540
fds = 0x7ffff5a3e540
recv_fds = 0x7ffff5a3e568
max_queue = 100
commandtimeout = 5
#2 0x00007ff480481062 in start_thread (arg=0x7ff4651b9700) at
pthread_create.c:312
__res = <optimized out>
pd = 0x7ff4651b9700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140687645054720, -492852880116455102, 0, 140688118599776, 17513280,
140687645054720,
486884492068350274, 486398082292607298}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev =
0x0, cleanup = 0x0,
canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#3 0x00007ff4801b4c1d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
Then I quit gdb:
(gdb) quit
A debugging session is active.
Inferior 1 [process 26126] will be detached.
Quit anyway? (y or n) y
Detaching from program: /usr/sbin/clamd, process 26126
If there is anything else you need me to do (such as install other
debug packages from some of the dependent libraries), let me know.
Regards,
Jim.
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
