Following a previous bug report about a stopping slapd server, during normal day work. As now we have a partially working slapd server, that answers some queries but ignores others. Using db5.1_stat and gdb was possible to get the following information:
1 - The BDB have an active dead lock, from db5.1_stat:
Locks grouped by lockers:
Locker Mode Count Status ----------------- Object ---------------
8002c7de dd=14 locks held 0 write locks 0 pid/thread
16486/140064000468736 priority 100
8002c7de READ 1 WAIT istPersonServices.bdb page 28
8002c7df dd=13 locks held 0 write locks 0 pid/thread
16486/140063983683328 priority 100
8002c7df READ 1 WAIT istPersonServices.bdb page 28
8002c7e0 dd=12 locks held 1 write locks 1 pid/thread
16486/140063992076032 priority 100
8002c7e0 WRITE 1 HELD istPersonServices.bdb page 28
(...)
2 - The thread 13 that owns the write lock has a very small stack
trace, from gdb:
Thread 13 (Thread 0x7f63307e5700 (LWP 22245)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00007f635ff2fc85 in ldap_int_thread_pool_wrapper (xpool=<optimized out>)
at ../../../../libraries/libldap_r/tpool.c:675
#2 0x00007f635e14e8ca in start_thread (arg=<optimized out>) at
pthread_create.c:300
#3 0x00007f635deb592d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4 0x0000000000000000 in ?? ()
3 - This slapd daemon is a slave and no longer receives updates
from the ldap master.
netstat -ntp
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
(...)
tcp 241009 0 XXX.XXX.XXX.XXX:49140 YYY.YYY.YYY.YYY:636
ESTABLISHED 16486/slapd
(...)
Attached is the full output from "db5.1_stat -CA" and is available on
request the full stack trace because of possible sensitive
information.
Jose Calhariz
--
--
Não há assunto pequeno. Há pequeno investidor
--Luiz da Câmara Cascudo
Default locking region information:
474 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
9 Number of lock modes
1500 Maximum number of locks possible
1500 Maximum number of lockers possible
1500 Maximum number of lock objects possible
40 Number of lock object partitions
60 Number of current locks
611 Maximum number of locks at any one time
33 Maximum number of locks in any one bucket
0 Maximum number of locks stolen by for an empty partition
0 Maximum number of locks stolen for any one partition
60 Number of current lockers
60 Maximum number of lockers at any one time
27 Number of current lock objects
314 Maximum number of lock objects at any one time
4 Maximum number of lock objects in any one bucket
0 Maximum number of objects stolen by for an empty partition
0 Maximum number of objects stolen for any one partition
912M Total number of locks requested (912517371)
912M Total number of locks released (912515767)
0 Total number of locks upgraded
449 Total number of locks downgraded
178955 Lock requests not available due to conflicts, for which we waited
1542 Lock requests not available due to conflicts, for which we did not wait
1 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
1MB 240KB The size of the lock region
613309 The number of partition locks that required waiting (0%)
24411 The maximum number of times any partition lock was waited for (0%)
15 The number of object queue operations that required waiting (0%)
940466 The number of locker allocations that required waiting (0%)
0 The number of region locks that required waiting (0%)
4 Maximum hash bucket length
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock REGINFO information:
Lock Region type
5 Region ID
__db.005 Region name
0x7f52215ba000 Region address
0x7f52215ba138 Region primary address
0 Region maximum allocation
0 Region allocated
Region allocations: 87 allocations, 0 failures, 0 frees, 1 longest
Allocations by power-of-two sizes:
1KB 2
2KB 0
4KB 41
8KB 40
16KB 0
32KB 0
64KB 2
128KB 0
256KB 2
512KB 0
1024KB 0
REGION_JOIN_OK Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock region parameters:
287 Lock region region mutex [0/94207 0% 18101/139990739408640]
2053 locker table size
2053 object table size
952 obj_off
218128 locker_off
0 need_dd
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock conflict matrix:
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by lockers:
Locker Mode Count Status ----------------- Object ---------------
8002c7de dd=14 locks held 0 write locks 0 pid/thread
16486/140064000468736 priority 100
8002c7de READ 1 WAIT istPersonServices.bdb page 28
8002c7df dd=13 locks held 0 write locks 0 pid/thread
16486/140063983683328 priority 100
8002c7df READ 1 WAIT istPersonServices.bdb page 28
8002c7e0 dd=12 locks held 1 write locks 1 pid/thread
16486/140063992076032 priority 100
8002c7e0 WRITE 1 HELD istPersonServices.bdb page 28
8002c7e1 dd=11 locks held 0 write locks 0 pid/thread
16486/140064008861440 priority 100
8002c7e1 READ 1 WAIT istPersonServices.bdb page 28
8002c7e2 dd=10 locks held 0 write locks 0 pid/thread
16486/140063908165376 priority 100
1c3 dd=58 locks held 1 write locks 0 pid/thread
16486/140064792749824 priority 100
1c3 READ 1 HELD id2entry.bdb handle 0
1c4 dd=57 locks held 1 write locks 0 pid/thread
16486/140064792749824 priority 100
1c4 READ 1 HELD dn2id.bdb handle 0
1c5 dd=56 locks held 0 write locks 0 pid/thread
16486/140064792749824 priority 100
1c6 dd=52 locks held 1 write locks 0 pid/thread
16486/140064679589632 priority 100
1c6 READ 1 HELD objectClass.bdb handle 0
1c7 dd=51 locks held 1 write locks 0 pid/thread
16486/140064679589632 priority 100
1c7 READ 1 HELD uid.bdb handle 0
1c8 dd=49 locks held 1 write locks 0 pid/thread
16486/140064679589632 priority 100
1c8 READ 1 HELD istPersonMailAlias.bdb handle 0
1c9 dd=48 locks held 1 write locks 0 pid/thread
16486/140064652273408 priority 100
1c9 READ 1 HELD istPersonServices.bdb handle 0
1ca dd=46 locks held 1 write locks 0 pid/thread
16486/140064627099392 priority 100
1ca READ 1 HELD uidNumber.bdb handle 0
1cb dd=44 locks held 1 write locks 0 pid/thread
16486/140064512272128 priority 100
1cb READ 1 HELD memberUid.bdb handle 0
1cc dd=41 locks held 1 write locks 0 pid/thread
16486/140064652273408 priority 100
1cc READ 1 HELD sambaSID.bdb handle 0
1cd dd=38 locks held 1 write locks 0 pid/thread
16486/140064601925376 priority 100
1cd READ 1 HELD uniqueMember.bdb handle 0
1ce dd=36 locks held 1 write locks 0 pid/thread
16486/140064601925376 priority 100
1ce READ 1 HELD krbPrincipalName.bdb handle 0
1cf dd=35 locks held 1 write locks 0 pid/thread
16486/140064319338240 priority 100
1cf READ 1 HELD istPersonUsername.bdb handle 0
1d0 dd=34 locks held 1 write locks 0 pid/thread
16486/140064652273408 priority 100
1d0 READ 1 HELD mail.bdb handle 0
1d1 dd=33 locks held 1 write locks 0 pid/thread
16486/140064627099392 priority 100
1d1 READ 1 HELD gidNumber.bdb handle 0
1d2 dd=32 locks held 1 write locks 0 pid/thread
16486/140064601925376 priority 100
1d2 READ 1 HELD cn.bdb handle 0
1d3 dd=31 locks held 1 write locks 0 pid/thread
16486/140064679589632 priority 100
1d3 READ 1 HELD entryUUID.bdb handle 0
1d4 dd=30 locks held 1 write locks 0 pid/thread
16486/140064679589632 priority 100
1d4 READ 1 HELD entryCSN.bdb handle 0
1d5 dd=29 locks held 1 write locks 0 pid/thread
16486/140064679589632 priority 100
1d5 READ 1 HELD sn.bdb handle 0
1d6 dd=28 locks held 1 write locks 0 pid/thread
16486/140064679589632 priority 100
1d6 READ 1 HELD istPersonIDNumber.bdb handle 0
1d7 dd=27 locks held 1 write locks 0 pid/thread
16486/140064679589632 priority 100
1d7 READ 1 HELD istPersonIDType.bdb handle 0
1d8 dd=26 locks held 1 write locks 0 pid/thread
16486/140064627099392 priority 100
1d8 READ 1 HELD krbPwdPolicyReference.bdb handle 0
1d9 dd=21 locks held 1 write locks 0 pid/thread
16486/140064193468160 priority 100
1d9 READ 1 HELD istPersonInternalEmail.bdb handle 0
1da dd=18 locks held 1 write locks 0 pid/thread
16486/140064126338816 priority 100
1da READ 1 HELD istPersonCertificateDN.bdb handle 0
8002c1d7 dd=55 locks held 0 write locks 0 pid/thread
16486/140064792749824 priority 100
8002c1d8 dd=54 locks held 0 write locks 0 pid/thread
16486/140064792749824 priority 100
8002c1d9 dd=53 locks held 0 write locks 0 pid/thread
16486/140064679589632 priority 100
8002c1d9 READ 1 WAIT istPersonServices.bdb page 28
8002c1dc dd=50 locks held 0 write locks 0 pid/thread
16486/140064652273408 priority 100
8002c1dc READ 1 WAIT istPersonServices.bdb page 28
8002c1df dd=47 locks held 0 write locks 0 pid/thread
16486/140064627099392 priority 100
8002c1df READ 1 WAIT istPersonServices.bdb page 28
8002c1e1 dd=45 locks held 0 write locks 0 pid/thread
16486/140064512272128 priority 100
8002c1e1 READ 1 WAIT istPersonServices.bdb page 28
8002c1e3 dd=43 locks held 0 write locks 0 pid/thread
16486/140064601925376 priority 100
8002c1e3 READ 1 WAIT istPersonServices.bdb page 28
8002c1e4 dd=42 locks held 0 write locks 0 pid/thread
16486/140064520664832 priority 100
8002c1e4 READ 1 WAIT istPersonServices.bdb page 28
8002c1e6 dd=40 locks held 0 write locks 0 pid/thread
16486/140064470316800 priority 100
8002c1e6 READ 1 WAIT istPersonServices.bdb page 28
8002c1e7 dd=39 locks held 0 write locks 0 pid/thread
16486/140064310945536 priority 100
8002c1e7 READ 1 WAIT istPersonServices.bdb page 28
8002c1e9 dd=37 locks held 0 write locks 0 pid/thread
16486/140064319338240 priority 100
8002c1e9 READ 1 WAIT istPersonServices.bdb page 28
8002e24b dd= 9 locks held 1 write locks 0 pid/thread
16486/140063908165376 priority 100
8002e24b READ 1 HELD 0x7a778 len: 9 data: 2c9901000000000000
8002e24c dd= 9 locks held 4 write locks 2 pid/thread
16486/140063908165376 priority 100
8002e24c READ 1 WAIT istPersonServices.bdb page 28
8002e24c WRITE 1 HELD istPersonServices.bdb page 2
8002e24c READ 1 HELD istPersonServices.bdb page 2
8002e24c WRITE 2 HELD entryCSN.bdb page 595
8002e24c READ 1 HELD entryCSN.bdb page 595
8002e24d dd= 8 locks held 0 write locks 0 pid/thread
16486/140063882991360 priority 100
8002e24d READ 1 WAIT istPersonServices.bdb page 28
8002e24e dd= 7 locks held 0 write locks 0 pid/thread
16486/140063782467328 priority 100
8002e24e READ 1 WAIT istPersonServices.bdb page 28
8002e24f dd= 6 locks held 0 write locks 0 pid/thread
16486/140063757293312 priority 100
8002e24f READ 1 WAIT istPersonServices.bdb page 28
8002e250 dd= 5 locks held 0 write locks 0 pid/thread
16486/140063732119296 priority 100
8002e250 READ 1 WAIT istPersonServices.bdb page 28
8002e251 dd= 4 locks held 0 write locks 0 pid/thread
16486/140063706945280 priority 100
8002e251 READ 1 WAIT istPersonServices.bdb page 28
8002e252 dd= 3 locks held 0 write locks 0 pid/thread
16486/140063681771264 priority 100
8002e252 READ 1 WAIT istPersonServices.bdb page 28
8002e253 dd= 2 locks held 0 write locks 0 pid/thread
16486/140063656597248 priority 100
8002e253 READ 1 WAIT istPersonServices.bdb page 28
8002e254 dd= 1 locks held 0 write locks 0 pid/thread
16486/140063631423232 priority 100
8002e254 READ 1 WAIT istPersonServices.bdb page 28
8002e255 dd= 0 locks held 0 write locks 0 pid/thread
16486/140063606249216 priority 100
8002e255 READ 1 WAIT istPersonServices.bdb page 28
8002c26b dd=25 locks held 0 write locks 0 pid/thread
16486/140064235427584 priority 100
8002c26b READ 1 WAIT istPersonServices.bdb page 28
8002c26c dd=24 locks held 0 write locks 0 pid/thread
16486/140064193468160 priority 100
8002c26c READ 1 WAIT istPersonServices.bdb page 28
8002c26d dd=23 locks held 0 write locks 0 pid/thread
16486/140064201860864 priority 100
8002c26d READ 1 WAIT istPersonServices.bdb page 28
8002c26e dd=22 locks held 0 write locks 0 pid/thread
16486/140064210253568 priority 100
8002c26e READ 1 WAIT istPersonServices.bdb page 28
8002c304 dd=20 locks held 0 write locks 0 pid/thread
16486/140064126338816 priority 100
8002c304 READ 1 WAIT istPersonServices.bdb page 28
8002c305 dd=19 locks held 0 write locks 0 pid/thread
16486/140064134731520 priority 100
8002c305 READ 1 WAIT istPersonServices.bdb page 28
8002c707 dd=17 locks held 0 write locks 0 pid/thread
16486/140064067598080 priority 100
8002c707 READ 1 WAIT istPersonServices.bdb page 28
8002c708 dd=16 locks held 0 write locks 0 pid/thread
16486/140064084383488 priority 100
8002c708 READ 1 WAIT istPersonServices.bdb page 28
8002c709 dd=15 locks held 0 write locks 0 pid/thread
16486/140064075990784 priority 100
8002c709 READ 1 WAIT istPersonServices.bdb page 28
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by object:
Locker Mode Count Status ----------------- Object ---------------
1d1 READ 1 HELD gidNumber.bdb handle 0
1c6 READ 1 HELD objectClass.bdb handle 0
1d0 READ 1 HELD mail.bdb handle 0
1d8 READ 1 HELD krbPwdPolicyReference.bdb handle 0
1c4 READ 1 HELD dn2id.bdb handle 0
8002e24c READ 1 HELD entryCSN.bdb page 595
8002e24c WRITE 2 HELD entryCSN.bdb page 595
1cb READ 1 HELD memberUid.bdb handle 0
8002e24b READ 1 HELD 0x7a778 len: 9 data: 2c9901000000000000
1d3 READ 1 HELD entryUUID.bdb handle 0
1cc READ 1 HELD sambaSID.bdb handle 0
1ca READ 1 HELD uidNumber.bdb handle 0
1d4 READ 1 HELD entryCSN.bdb handle 0
1d5 READ 1 HELD sn.bdb handle 0
1d7 READ 1 HELD istPersonIDType.bdb handle 0
1c3 READ 1 HELD id2entry.bdb handle 0
1c8 READ 1 HELD istPersonMailAlias.bdb handle 0
1d2 READ 1 HELD cn.bdb handle 0
1d9 READ 1 HELD istPersonInternalEmail.bdb handle 0
1ce READ 1 HELD krbPrincipalName.bdb handle 0
1cd READ 1 HELD uniqueMember.bdb handle 0
1c7 READ 1 HELD uid.bdb handle 0
1da READ 1 HELD istPersonCertificateDN.bdb handle 0
1cf READ 1 HELD istPersonUsername.bdb handle 0
1d6 READ 1 HELD istPersonIDNumber.bdb handle 0
8002c7e0 WRITE 1 HELD istPersonServices.bdb page 28
8002e24c READ 1 WAIT istPersonServices.bdb page 28
8002c708 READ 1 WAIT istPersonServices.bdb page 28
8002c304 READ 1 WAIT istPersonServices.bdb page 28
8002c1e7 READ 1 WAIT istPersonServices.bdb page 28
8002c707 READ 1 WAIT istPersonServices.bdb page 28
8002c7de READ 1 WAIT istPersonServices.bdb page 28
8002c7e1 READ 1 WAIT istPersonServices.bdb page 28
8002c1e1 READ 1 WAIT istPersonServices.bdb page 28
8002c1dc READ 1 WAIT istPersonServices.bdb page 28
8002c709 READ 1 WAIT istPersonServices.bdb page 28
8002c7df READ 1 WAIT istPersonServices.bdb page 28
8002c1e4 READ 1 WAIT istPersonServices.bdb page 28
8002c26b READ 1 WAIT istPersonServices.bdb page 28
8002c305 READ 1 WAIT istPersonServices.bdb page 28
8002c26c READ 1 WAIT istPersonServices.bdb page 28
8002c26e READ 1 WAIT istPersonServices.bdb page 28
8002c1e6 READ 1 WAIT istPersonServices.bdb page 28
8002c1e9 READ 1 WAIT istPersonServices.bdb page 28
8002c1e3 READ 1 WAIT istPersonServices.bdb page 28
8002c1d9 READ 1 WAIT istPersonServices.bdb page 28
8002c26d READ 1 WAIT istPersonServices.bdb page 28
8002c1df READ 1 WAIT istPersonServices.bdb page 28
8002e24d READ 1 WAIT istPersonServices.bdb page 28
8002e24e READ 1 WAIT istPersonServices.bdb page 28
8002e250 READ 1 WAIT istPersonServices.bdb page 28
8002e24f READ 1 WAIT istPersonServices.bdb page 28
8002e251 READ 1 WAIT istPersonServices.bdb page 28
8002e252 READ 1 WAIT istPersonServices.bdb page 28
8002e253 READ 1 WAIT istPersonServices.bdb page 28
8002e254 READ 1 WAIT istPersonServices.bdb page 28
8002e255 READ 1 WAIT istPersonServices.bdb page 28
1c9 READ 1 HELD istPersonServices.bdb handle 0
8002e24c READ 1 HELD istPersonServices.bdb page 2
8002e24c WRITE 1 HELD istPersonServices.bdb page 2
signature.asc
Description: Digital signature
