On 8/9/23 12:05, Thierry Bordaz wrote:
On 8/9/23 18:55, Harry G Coin wrote:
Theirry asked for a recap summary below, so forgive the 'top post'.
Here it is:
4.9.10 default install on two systems call them primary (with
kasp.db) and secondary but otherwise multi-master, 1g link between
them, modest/old cpu, drives, 5Gmemory, with dns/dnssec and adtrust
(aimed at local samba share support only). Unremarkable initial
install. Normal operations, GUI, etc.
A python program using the ldap2 backend on Primary starts loading a
few dozen default domains with A / AAAA and associated PTR records.
It first does dns find/show to check for existence, and if absent
adds the domain/subdomain, missing A / AAAA assoc PTR etc.
Extensive traffic in the logs to do with dnssec, notifies being sent
back and forth between primary and secondary by bind9 (which you'd
think already had the info in ldap so why 'notify' via bind really?)
serial numbers going up, dnssec updates. Every now and then the
program checks whether dnssec keys need rotating or if new zones
appear, but that's fairly infrequent and seems unrelated.
After not more than a few minutes of adding records, in Primary's log
"writeback to ldap failed" will appear. There will be nothing in
any log indicating anything else amiss, 'systemctl is-system-running'
reports 'running'. Login attempts on the GUI fail 'for an unknown
reason', named/bind9 queries for A/AAAA seem to work. Anything that
calls ns-slapd times out or hangs waiting forever. CPU usage near 0.
Did you get a pstack (ns-slapd) at that time ?
Yes, posted 8/8/23 and again now:
[root@registry2 ~]# pstack 1405 > ns-slapd3.log
[root@registry2 ~]# more ns-slapd3.log
Thread 33 (Thread 0x7f66366f5700 (LWP 2654)):
#0 0x00007f6639b6d455 in pthread_rwlock_wrlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d380 in map_wrlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e8d393 in backend_shr_post_delete_cb.part () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f6628e8d508 in backend_shr_betxn_post_delete_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#4 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c8200,
operation=operation@entry=563, pb=pb@entry=0x7f65fbdffcc0,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#5 0x00007f663d7beec4 in plugin_call_list (pb=0x7f65fbdffcc0,
operation=563, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#6 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f65fbdffcc0, whichfunction=whichfunction@entry=563) at
ldap/servers/slapd/plugin.c:442
#7 0x00007f662ae6ac83 in ldbm_back_delete (pb=0x7f65fbdffcc0) at
ldap/servers/slapd/back-ldbm/ldbm_delete.c:1289
#8 0x00007f663d7696ac in op_shared_delete (pb=pb@entry=0x7f65fbdffcc0)
at ldap/servers/slapd/delete.c:338
#9 0x00007f663d7698bd in delete_internal_pb
(pb=pb@entry=0x7f65fbdffcc0) at ldap/servers/slapd/delete.c:209
#10 0x00007f663d769b3b in slapi_delete_internal_pb
(pb=pb@entry=0x7f65fbdffcc0) at ldap/servers/slapd/delete.c:151
#11 0x00007f66294c4fde in delete_changerecord (cnum=cnum@entry=27941) at
ldap/servers/plugins/retrocl/retrocl_trim.c:89
#12 0x00007f66294c51a1 in trim_changelog () at
ldap/servers/plugins/retrocl/retrocl_trim.c:290
#13 0x00007f66294c51a1 in changelog_trim_thread_fn (arg=<optimized out>)
at ldap/servers/plugins/retrocl/retrocl_trim.c:333
#14 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#15 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#16 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 32 (Thread 0x7f65f61fc700 (LWP 1438)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f6600e07580,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f6600e07580,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f6600e07580, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f6600e07580) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f6600e07580) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f6600e07580, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 31 (Thread 0x7f65f69fd700 (LWP 1437)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f6601c1a000,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f6601c1a000,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f6601c1a000, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f6601c1a000) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f6601c1a000) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f6601c1a000, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 30 (Thread 0x7f65f71fe700 (LWP 1436)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f6606438580,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f6606438580,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f6606438580, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f6606438580) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f6606438580) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f6606438580, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 29 (Thread 0x7f65f7bff700 (LWP 1435)):
#0 0x00007f6639b6e45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f663a1c7e3b in PR_EnterMonitor () at target:/lib64/libnspr4.so
#2 0x00007f662ae4a394 in dblayer_lock_backend (be=<optimized out>) at
ldap/servers/slapd/back-ldbm/dblayer.c:819
#3 0x00007f662ae4a46e in dblayer_txn_begin (be=0x7f6628285e80,
parent_txn=0x0, txn=txn@entry=0x7f65f7bfd8c8) at
ldap/servers/slapd/back-ldbm/dblayer.c:695
#4 0x00007f662ae7773a in ldbm_back_modify (pb=<optimized out>) at
ldap/servers/slapd/back-ldbm/ldbm_modify.c:672
#5 0x00007f663d7aa300 in op_shared_modify (pb=pb@entry=0x7f6606a08580,
pw_change=pw_change@entry=0, old_pw=0x0) at
ldap/servers/slapd/modify.c:1025
#6 0x00007f663d7aba00 in do_modify (pb=pb@entry=0x7f6606a08580) at
ldap/servers/slapd/modify.c:380
#7 0x000055a71640175b in connection_dispatch_operation
(pb=0x7f6606a08580, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:651
#8 0x000055a71640175b in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#9 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#10 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#11 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 28 (Thread 0x7f65f89ff700 (LWP 1434)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f66263ee3c0,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f66263ee3c0,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f66263ee3c0, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f66263ee3c0) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f66263ee3c0) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f66263ee3c0, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 27 (Thread 0x7f65f95fe700 (LWP 1433)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f6608806580,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f6608806580,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f6608806580, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f6608806580) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f6608806580) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f6608806580, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 26 (Thread 0x7f65f9fff700 (LWP 1432)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f6609605000,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f6609605000,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f6609605000, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f6609605000) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f6609605000) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f6609605000, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 25 (Thread 0x7f65fb1ff700 (LWP 1431)):
#0 0x00007f6639b6e45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f663a1c7e3b in PR_EnterMonitor () at target:/lib64/libnspr4.so
#2 0x00007f662ae4a394 in dblayer_lock_backend (be=<optimized out>) at
ldap/servers/slapd/back-ldbm/dblayer.c:819
#3 0x00007f662ae4a46e in dblayer_txn_begin (be=0x7f66282d8f00,
parent_txn=0x7f65f963b300, txn=txn@entry=0x7f65fb1fd028) at
ldap/servers/slapd/back-ldbm/dblayer.c:695
#4 0x00007f662ae5b7c1 in ldbm_back_add (pb=0x7f65f72ca820) at
ldap/servers/slapd/back-ldbm/ldbm_add.c:269
#5 0x00007f663d759728 in op_shared_add (pb=pb@entry=0x7f65f72ca820) at
ldap/servers/slapd/add.c:692
#6 0x00007f663d759bbe in add_internal_pb (pb=pb@entry=0x7f65f72ca820)
at ldap/servers/slapd/add.c:407
#7 0x00007f663d75a975 in slapi_add_internal_pb
(pb=pb@entry=0x7f65f72ca820) at ldap/servers/slapd/add.c:331
#8 0x00007f66294c48bf in write_replog_db (newsuperior=0x0,
modrdn_mods=0x0, newrdn=0x0, post_entry=<optimized out>,
log_e=0x7f65f96d7a40, curtime=1691451921, flag=0, log_m=0x7f6628278a40,
dn=0x7f65f961b4f0
"idnsname=112.12.10.in-addr.arpa.,cn=dns,dc=1,dc=quietfountain,d
c=com", optype=<optimized out>, pb=0x7f65f95ff000) at
ldap/servers/plugins/retrocl/retrocl_po.c:369
#9 0x00007f66294c48bf in retrocl_postob (pb=0x7f65f95ff000,
optype=<optimized out>) at ldap/servers/plugins/retrocl/retrocl_po.c:697
#10 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324aae00,
operation=operation@entry=561, pb=pb@entry=0x7f65f95ff000,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#11 0x00007f663d7beec4 in plugin_call_list (pb=0x7f65f95ff000,
operation=561, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#12 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f65f95ff000, whichfunction=whichfunction@entry=561) at
ldap/servers/slapd/plugin.c:442
#13 0x00007f662ae78658 in ldbm_back_modify (pb=<optimized out>) at
ldap/servers/slapd/back-ldbm/ldbm_modify.c:1002
#14 0x00007f663d7aa300 in op_shared_modify (pb=pb@entry=0x7f65f95ff000,
pw_change=pw_change@entry=0, old_pw=0x0) at
ldap/servers/slapd/modify.c:1025
#15 0x00007f663d7aba00 in do_modify (pb=pb@entry=0x7f65f95ff000) at
ldap/servers/slapd/modify.c:380
#16 0x000055a71640175b in connection_dispatch_operation
(pb=0x7f65f95ff000, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:651
#17 0x000055a71640175b in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#18 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#19 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#20 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 24 (Thread 0x7f65fbdfe700 (LWP 1430)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f65f71ff000,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f65f71ff000,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f65f71ff000, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f65f71ff000) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f65f71ff000) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f65f71ff000, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 23 (Thread 0x7f65fc7ff700 (LWP 1429)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f65f7fff000,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f65f7fff000,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f65f7fff000, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f65f7fff000) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f65f7fff000) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f65f7fff000, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 22 (Thread 0x7f65fd3ff700 (LWP 1428)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f6609cf1580,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f6609cf1580,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f6609cf1580, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f6609cf1580) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f6609cf1580) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f6609cf1580, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 21 (Thread 0x7f65fe9ff700 (LWP 1427)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f65fa200000,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f65fa200000,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f65fa200000, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f65fa200000) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f65fa200000) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f65fa200000, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 20 (Thread 0x7f65ff5fc700 (LWP 1426)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f65fa400000,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f65fa400000,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f65fa400000, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f65fa400000) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f65fa400000) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f65fa400000, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 19 (Thread 0x7f65ffdfd700 (LWP 1425)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f65fbdff000,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f65fbdff000,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f65fbdff000, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f65fbdff000) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f65fbdff000) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f65fbdff000, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 18 (Thread 0x7f66005fe700 (LWP 1424)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f65fdfff000,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f65fdfff000,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f65fdfff000, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f65fdfff000) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f65fdfff000) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f65fdfff000, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 17 (Thread 0x7f6600dff700 (LWP 1423)):
#0 0x00007f6639b6d022 in pthread_rwlock_rdlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d242 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e88298 in backend_bind_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c7300,
operation=operation@entry=401, pb=pb@entry=0x7f65fddff000,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#4 0x00007f663d7beec4 in plugin_call_list (pb=0x7f65fddff000,
operation=401, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#5 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f65fddff000, whichfunction=whichfunction@entry=401) at
ldap/servers/slapd/plugin.c:442
#6 0x000055a716413add in ids_sasl_check_bind
(pb=pb@entry=0x7f65fddff000) at ldap/servers/slapd/saslbind.c:1205
#7 0x000055a7163fbd27 in do_bind (pb=pb@entry=0x7f65fddff000) at
ldap/servers/slapd/bind.c:367
#8 0x000055a716401835 in connection_dispatch_operation
(pb=0x7f65fddff000, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:626
#9 0x000055a716401835 in connection_threadmain (arg=<optimized out>) at
ldap/servers/slapd/connection.c:1803
#10 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#11 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#12 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 16 (Thread 0x7f660295d700 (LWP 1421)):
#0 0x00007f6639b6d455 in pthread_rwlock_wrlock () at
target:/lib64/libpthread.so.0
#1 0x00007f6628e9d380 in map_wrlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#2 0x00007f6628e8f54a in backend_shr_post_modify_cb.part () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3 0x00007f6628e8fd48 in backend_shr_betxn_post_modify_cb () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#4 0x00007f663d7bec79 in plugin_call_func (list=0x7f66324c8200,
operation=operation@entry=561, pb=pb@entry=0x7f6626480ca0,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#5 0x00007f663d7beec4 in plugin_call_list (pb=0x7f6626480ca0,
operation=561, list=<optimized out>) at ldap/servers/slapd/plugin.c:1973
#6 0x00007f663d7beec4 in plugin_call_plugins
(pb=pb@entry=0x7f6626480ca0, whichfunction=whichfunction@entry=561) at
ldap/servers/slapd/plugin.c:442
#7 0x00007f663d773256 in dse_modify (pb=0x7f6626480ca0) at
ldap/servers/slapd/dse.c:2048
#8 0x00007f663d7aa300 in op_shared_modify (pb=pb@entry=0x7f6626480ca0,
pw_change=pw_change@entry=0, old_pw=0x0) at
ldap/servers/slapd/modify.c:1025
#9 0x00007f663d7aaf4b in modify_internal_pb
(pb=pb@entry=0x7f6626480ca0) at ldap/servers/slapd/modify.c:575
#10 0x00007f663d7abbdb in slapi_modify_internal_pb
(pb=pb@entry=0x7f6626480ca0) at ldap/servers/slapd/modify.c:445
#11 0x00007f662a595bf8 in replica_update_state (when=<optimized out>,
arg=<optimized out>) at
ldap/servers/plugins/replication/repl5_replica.c:2845
#12 0x00007f663d77f1dc in eq_call_all_rel () at
ldap/servers/slapd/eventq.c:278
#13 0x00007f663d77f1dc in eq_loop_rel (arg=<optimized out>) at
ldap/servers/slapd/eventq.c:324
#14 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#15 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#16 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 15 (Thread 0x7f660315e700 (LWP 1420)):
#0 0x00007f6639b6e45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f663a1c7ba8 in PR_WaitCondVar () at target:/lib64/libnspr4.so
#2 0x00007f663d77fa23 in eq_loop (arg=<optimized out>) at
ldap/servers/slapd/eventq-deprecated.c:322
#3 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 14 (Thread 0x7f6604160700 (LWP 1418)):
#0 0x00007f6639b6e45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f663d7e573d in slapi_wait_condvar_pt (cvar=0x7f6625957be0,
mutex=0x7f6625957940, timeout=timeout@entry=0x0) at
ldap/servers/slapd/slapi2runtime.c:179
#2 0x00007f66292b7af1 in roles_cache_wait_on_change
(arg=0x7f66264ad880) at ldap/servers/plugins/roles/roles_cache.c:387
#3 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 13 (Thread 0x7f6604961700 (LWP 1417)):
#0 0x00007f6639b6e45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f663d7e573d in slapi_wait_condvar_pt (cvar=0x7f6625957970,
mutex=0x7f6625957b80, timeout=timeout@entry=0x0) at
ldap/servers/slapd/slapi2runtime.c:179
#2 0x00007f66292b7af1 in roles_cache_wait_on_change
(arg=0x7f66263d3800) at ldap/servers/plugins/roles/roles_cache.c:387
#3 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 12 (Thread 0x7f6605162700 (LWP 1416)):
#0 0x00007f6639b6e45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f663d7e573d in slapi_wait_condvar_pt (cvar=0x7f66259575e0,
mutex=0x7f6625957a60, timeout=timeout@entry=0x0) at
ldap/servers/slapd/slapi2runtime.c:179
#2 0x00007f66292b7af1 in roles_cache_wait_on_change
(arg=0x7f6625952f00) at ldap/servers/plugins/roles/roles_cache.c:387
#3 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 11 (Thread 0x7f66073fe700 (LWP 1415)):
#0 0x00007f6639b6e848 in pthread_cond_timedwait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f662a5863be in protocol_sleep (prp=prp@entry=0x7f6626d02f00,
duration=duration@entry=300) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:1222
#2 0x00007f662a589fb5 in repl5_inc_run (prp=0x7f6626d02f00) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:779
#3 0x00007f662a58e7e1 in prot_thread_main (arg=0x7f662563e5b0) at
ldap/servers/plugins/replication/repl5_protocol.c:252
#4 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#5 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#6 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 10 (Thread 0x7f6607ffe700 (LWP 1414)):
#0 0x00007f6639b6e45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f662a5863f3 in protocol_sleep (prp=prp@entry=0x7f6626d02d00,
duration=duration@entry=0) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:1224
#2 0x00007f662a589880 in repl5_inc_run (prp=0x7f6626d02d00) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:851
#3 0x00007f662a58e7e1 in prot_thread_main (arg=0x7f662563e510) at
ldap/servers/plugins/replication/repl5_protocol.c:252
#4 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#5 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#6 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 9 (Thread 0x7f663e029700 (LWP 1413)):
#0 0x00007f6639b6e7aa in pthread_cond_timedwait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f663a1c7673 in pt_TimedWait () at target:/lib64/libnspr4.so
#2 0x00007f663a1c7b3e in PR_WaitCondVar () at target:/lib64/libnspr4.so
#3 0x00007f662a56e91e in _cl5TrimMain (param=<optimized out>) at
ldap/servers/plugins/replication/cl5_api.c:3189
#4 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#5 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#6 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 8 (Thread 0x7f66087ff700 (LWP 1412)):
#0 0x00007f663befcf41 in poll () at target:/lib64/libc.so.6
#1 0x00007f662d3e1b8f in ipa_cldap_worker () at
target:/usr/lib64/dirsrv/plugins/libipa_cldap.so
#2 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#3 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 7 (Thread 0x7f66095ff700 (LWP 1411)):
#0 0x00007f6639b6e45c in pthread_cond_wait@@GLIBC_2.3.2 () at
target:/lib64/libpthread.so.0
#1 0x00007f663d7e573d in slapi_wait_condvar_pt (cvar=0x7f6626ce72d0,
mutex=0x7f6626ce71b0, timeout=timeout@entry=0x0) at
ldap/servers/slapd/slapi2runtime.c:179
#2 0x00007f6630084659 in cos_cache_wait_on_change (arg=<optimized out>)
at ldap/servers/plugins/cos/cos_cache.c:404
#3 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 6 (Thread 0x7f660a5ff700 (LWP 1410)):
#0 0x00007f663beff67f in select () at target:/lib64/libc.so.6
#1 0x00007f663d7fa5b4 in DS_Sleep (ticks=<optimized out>) at
ldap/servers/slapd/util.c:1060
#2 0x00007f662ae90afc in locks_monitoring_threadmain
(param=0x7f6636249700) at
ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:2991
#3 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 5 (Thread 0x7f660b082700 (LWP 1409)):
#0 0x00007f663beff67f in select () at target:/lib64/libc.so.6
#1 0x00007f663d7fa5b4 in DS_Sleep (ticks=<optimized out>) at
ldap/servers/slapd/util.c:1060
#2 0x00007f662ae84408 in perfctrs_wait
(milliseconds=milliseconds@entry=1000, priv=<optimized out>,
db_env=<optimized out>) at ldap/servers/slapd/back-ldbm/perfctrs.c:80
#3 0x00007f662ae90994 in perf_threadmain (param=0x7f6636249700) at
ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:2916
#4 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#5 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#6 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 4 (Thread 0x7f660b883700 (LWP 1408)):
#0 0x00007f663beff67f in select () at target:/lib64/libc.so.6
#1 0x00007f663d7fa5b4 in DS_Sleep (ticks=ticks@entry=250) at
ldap/servers/slapd/util.c:1060
#2 0x00007f662ae90807 in trickle_threadmain (param=0x7f6636249700) at
ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3976
#3 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 3 (Thread 0x7f660c084700 (LWP 1407)):
#0 0x00007f663beff67f in select () at target:/lib64/libc.so.6
#1 0x00007f663d7fa5b4 in DS_Sleep (ticks=ticks@entry=2500) at
ldap/servers/slapd/util.c:1060
#2 0x00007f662ae92b43 in checkpoint_threadmain (param=0x7f6636249700)
at ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3824
#3 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 2 (Thread 0x7f660c885700 (LWP 1406)):
#0 0x00007f663beff67f in select () at target:/lib64/libc.so.6
#1 0x00007f663d7fa5b4 in DS_Sleep (ticks=ticks@entry=100) at
ldap/servers/slapd/util.c:1060
#2 0x00007f662ae902e7 in deadlock_threadmain (param=0x7f6636249700) at
ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:3494
#3 0x00007f663a1cd968 in _pt_root () at target:/lib64/libnspr4.so
#4 0x00007f6639b681ca in start_thread () at target:/lib64/libpthread.so.0
#5 0x00007f663be12e73 in clone () at target:/lib64/libc.so.6
Thread 1 (Thread 0x7f663e1ca4c0 (LWP 1405)):
#0 0x00007f663befcf41 in poll () at target:/lib64/libc.so.6
#1 0x00007f663a1c9658 in _pr_poll_with_poll () at
target:/lib64/libnspr4.so
#2 0x000055a7164051a6 in slapd_daemon (ports=<optimized out>) at
ldap/servers/slapd/daemon.c:1338
#3 0x000055a7163f8ded in main (argc=5, argv=0x7fffae95d2c8) at
ldap/servers/slapd/main.c:1099
'systemctl restart ipa' and or reboot restores operations-- HOWEVER
there will be at least a 10 minute wait with ns-slapd at 100% CPU
until the reboot process forcibly kills it.
I guess most ns-slapd workers (thread running the requests) have been
stopped but no idea which ones is eating CPU. A 'top -H' and pstack
would help.
Done! Above. Thanks!
Upgrading to 4.9.11 caused the 'writeback to ldap failed' message to
move to Secondary, not primary. Same further consequences.
Alexander's dsconf notion changed the appearance, it broke dnssec
updates with an LDAP timeout error message.
There is nothing whatever remarkable about this two node setup. I
suspect that test environments using the latest processors and all
nvme storage is just too performant to manifest it, or the test
environments don't have dnssec enabled and don't add a few thousand
records to a few dozen subdomains.
I need some way forward, it's dead in the water now. Presently my
'plan' such as it is -- is move freeipa VMs to faster systems with
more memory and 10gb interconnects in hopes of not hitting this, but
of course this is one of those 'sword hanging over everyone's head by
a thread' 'don't breathe on it wrong or you'll die' situations that
needs an answer before trust can come back.
I appreciate the focus!
On 8/9/23 11:24, Thierry Bordaz wrote:
On 8/9/23 17:15, Harry G Coin wrote:
On 8/9/23 01:00, Alexander Bokovoy wrote:
On Аўт, 08 жні 2023, Harry G Coin wrote:
Thanks for your help. Details below. The problem 'moved' in I
hope a diagnositcally useful way, but the system remains broken.
On 8/8/23 08:54, Alexander Bokovoy wrote:
On Аўт, 08 жні 2023, Harry G Coin wrote:
On 8/8/23 02:43, Alexander Bokovoy wrote:
pstack $(pgrep ns-slapd) > ns-slapd log
Tried an upgrade from 4.9.10 to 4.9.11, the "writeback to ldap
failed" error moved from the primary instance (on which the dns
records were being added) to the replica which hung in the same
fashion. Here's the log you asked for from attempting
'systemctl restart dirsrv@...' it just hangs at 100% cpu for
about 10 minutes.
Thank you. Are you using schema compat for some legacy clients?
This is a fresh install of 4.9.10 about a week ago, upgraded to
4.9.11 yesterday, just two freeipa instances and no appreciable
user load, using the install defaults. The 'in house' system then
starts loading lots of dns records via the python ldap2 interface
on the first of two systems installed, the replica produced what
you see in this post. There is no 'private' information involved
of any sort, it's supposed to field DNS calls from the public but
was so unreliable I had to implement unbound on other servers, so
all freeipa does is IXFR to unbound for the heavy load. I
suppose there may be <16 other in-house lab systems, maybe 2 or 3
with any activity, that use it for dns. The only other clue is
these are running on VMs in older servers and have no other
software packages installed other than freeipa and what freeipa
needs to run, and the in-house program that loads the dns.
Just to exclude potential problems with schema compat, it can be
disabled if you are not using it.
How? The installs just use all the defaults, other than enabling
dnssec and PTR records for all a/aaaa.
I'm officially in 'desperation mode' as not being able to populate
DNS in freeipa reduces everyone to pencil and paper and coffee with
full project stoppage until it's fixed or at least 'worked
around'. So anything that 'might help' can be sacrificed so at
least 'something' works 'somewhat'. If old AD needs to be
'broken' or 'off' but mostly the rest of it 'works sort of' then
how do I do it?
Really this can't be hard to reproduce, it's just two instances
with a 1G link between them, each with a pair of old rusty hard
drives in an lvm mirror using a COW file system, dnssec on, and one
of them loading lots of dns with reverse pointers for each A/AAAA
with maybe 200 to 600 PTR records per *arpa and maybe 10-200
records per subdomain, maybe 200 domains total. A couple python
for loops and hey presto you'll see freeipa lock up without notice
in your lab as well. I just can't imagine causing these race
conditions to appear in the case of the only important load being
DNS adds/finds/shows should be difficult.
I appreciate the help, and have become officially fearful about
freeipa. Maybe it's seldom used extensively for DNS and so my use
case is an outlier? Why are so few seeing this? It's a fully
default package install, no custom changes to the OS, freeipa,
other packages. I don't get it.
Thanks for any leads or help!
Hi Harry,
I agree with Mark, nothing suspicious on Thread30. It is flushing
its txn.
The discussion is quite long, do you mind to re-explain what are the
current symptoms ?
Is it hanging during update ? consuming CPU ?
Could you run top -H -p <pid> -n 5 -d 3
if it is hanging could you run 'db_stat -CA -h
/dev/shm/slapd-<inst>/ -N'
regards
thierry
I don't think it is about named per se, it is a bit of an unfortunate
interop inside ns-slapd between different plugins. bind-dyndb-ldap
relies on the syncrepl extension which implementation in ns-slapd is
using the retro changelog content. Retro changelog plugin triggers
some
updates that cause schema compatibility plugin to lock itself up
depending on the order of updates that retro changelog would
capture. We
fixed that in slapi-nis package some time ago and it *should* be
ignoring the retro changelog changes but somehow they still propagate
into it. There are few places in ns-slapd which were addressed just
recently and those updates might help (out later this year in RHEL).
Disabling schema compat would be the best.
What's worse, every reboot attempt waits the full '9 min 29 secs'
before systemd forcibly terminates ns-slapd to finish the 'stop
job'.
That's why I'm so troubled by all this, it's not like there is
any interference from anything other than what freeipa puts out
there, and it just locks with a message that gives no indication
of what to do about it, with nothing in any logs and 'systemctl
is-system-running' reports 'running'.
You could easily replicate this: imagine a simple validation
test that sets up two freeipa nodes, turns on dnssec, creates
some domains, then adds A AAAA and *.arpa records using the ldap2
api on one of the nodes. Maybe limit the net speed between the
nodes to a 1GB link typical, maybe at most 4 processor cores of
some older vintage and 5GB memory. It takes less than 2 minutes
after dns load start to lock up.
What's really odd is bind9 / named keeps blasting out change
notifications for some of the updated domains, then a few lines
later, with no intervening activity in any log or by any program
affecting the zone, will publish further change notifications
with a new serial number for the same zone. This happens for all
the zones that get modifications. I'm thinking 'rr'
computations? I wonder if those entries-- being auto-generated
internally -- are creating a 'flow control' issue between the
primary and replica.
This is something that retro changelog is responsible for as it is
the
data store used by the syncrepl protocol implementation. If these
'changes' appear again and again, it means retro changelog plugin
marks
them as new for this particular syncrepl client (bind-dyndb-ldap).
All threads other than the thread 30 are normal ones (idle
threads) but
this one blocks the database backend in the log flush sequence while
writing the retro changelog entry for this updated DNS record:
Thread 30 (Thread 0x7f0e583ff700 (LWP 1438)):
#0 0x00007f0e9bf7d8af in fdatasync () at target:/lib64/libc.so.6
#1 0x00007f0e91cbe6b5 in __os_fsync () at
target:/lib64/libdb-5.3.so
#2 0x00007f0e91ca598c in __log_flush_int () at
target:/lib64/libdb-5.3.so
#3 0x00007f0e91ca7dd0 in __log_flush () at
target:/lib64/libdb-5.3.so
#4 0x00007f0e91ca7f73 in __log_flush_pp () at
target:/lib64/libdb-5.3.so
#5 0x00007f0e8afe1304 in bdb_txn_commit (li=<optimized out>,
txn=0x7f0e583fd028, use_lock=1) at
ldap/servers/slapd/back-ldbm/db-bdb/bdb_layer.c:2772
#6 0x00007f0e8af95515 in dblayer_txn_commit (be=0x7f0e88424f00,
txn=<optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:736
#7 0x00007f0e8afa7ebe in ldbm_back_add (pb=0x7f0e85748860) at
ldap/servers/slapd/back-ldbm/ldbm_add.c:1242
#8 0x00007f0e9d7d7728 in op_shared_add
(pb=pb@entry=0x7f0e85748860) at ldap/servers/slapd/add.c:692
#9 0x00007f0e9d7d7bbe in add_internal_pb
(pb=pb@entry=0x7f0e85748860) at ldap/servers/slapd/add.c:407
#10 0x00007f0e9d7d8975 in slapi_add_internal_pb
(pb=pb@entry=0x7f0e85748860) at ldap/servers/slapd/add.c:331
#11 0x00007f0e8960f8bf in write_replog_db (newsuperior=0x0,
modrdn_mods=0x0, newrdn=0x0, post_entry=<optimized out>,
log_e=0x7f0e4df5b9c0, curtime=1691511446, flag=0,
log_m=0x7f0e5ec0d440, dn=0x7f0e57a40740
"idnsname=8.0.f.0.0.0.0.0.0.0.0.1.0.0.c.f.ip6.arpa.,cn=dns,dc=1,dc=quietfountain,dc=com",
optype=<optimized out>, pb=0x7f0e66a09580) at
ldap/servers/plugins/retrocl/retrocl_po.c:369 #12
0x00007f0e8960f8bf in retrocl_postob (pb=0x7f0e66a09580,
optype=<optimized out>) at
ldap/servers/plugins/retrocl/retrocl_po.c:697
#13 0x00007f0e9d83cc79 in plugin_call_func (list=0x7f0e924aae00,
operation=operation@entry=561, pb=pb@entry=0x7f0e66a09580,
call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2032
#14 0x00007f0e9d83cec4 in plugin_call_list (pb=0x7f0e66a09580,
operation=561, list=<optimized out>) at
ldap/servers/slapd/plugin.c:1973
#15 0x00007f0e9d83cec4 in plugin_call_plugins
(pb=pb@entry=0x7f0e66a09580,
whichfunction=whichfunction@entry=561) at
ldap/servers/slapd/plugin.c:442
#16 0x00007f0e8afc3658 in ldbm_back_modify (pb=<optimized out>)
at ldap/servers/slapd/back-ldbm/ldbm_modify.c:1002
#17 0x00007f0e9d828300 in op_shared_modify
(pb=pb@entry=0x7f0e66a09580, pw_change=pw_change@entry=0,
old_pw=0x0) at ldap/servers/slapd/modify.c:1025
#18 0x00007f0e9d829a00 in do_modify (pb=pb@entry=0x7f0e66a09580)
at ldap/servers/slapd/modify.c:380
#19 0x0000564ed703475b in connection_dispatch_operation
(pb=0x7f0e66a09580, op=<optimized out>, conn=<optimized out>) at
ldap/servers/slapd/connection.c:651
#20 0x0000564ed703475b in connection_threadmain (arg=<optimized
out>) at ldap/servers/slapd/connection.c:1803
#21 0x00007f0e9a24b968 in _pt_root () at target:/lib64/libnspr4.so
#22 0x00007f0e99be61ca in start_thread () at
target:/lib64/libpthread.so.0
#23 0x00007f0e9be90e73 in clone () at target:/lib64/libc.so.6
Mark, Thierry, any hints here? (For full trace see thread
https://lists.fedorahosted.org/archives/list/[email protected]/thread/TMRXHCORFU3QRQL6FSZTS4OIHYOAVXWF/)
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
