Had another crash. Attached is the log from " thread apply all bt full".
Regards, Suresh On Thu, Jan 16, 2025 at 7:16 PM Suresh Veliveli < [email protected]> wrote: > Any thoughts on this? > > Regards, > Suresh > > On Mon, Jan 13, 2025 at 10:42 AM Suresh Veliveli < > [email protected]> wrote: > >> Hi Ondřej, >> >> Attached is the file from the last crash for "thread apply all bt full". >> I built it from the src (openldap.org). The installation is prefixed to >> /var/services/openldap directory. I do have "stats sync" log level enabled. >> Our logs are huge, I could get the necessary info if you can tell what I >> need to look for. >> >> Thanks, >> Suresh >> >> On Mon, Jan 13, 2025 at 7:31 AM Ondřej Kuzník <[email protected]> >> wrote: >> >>> On Thu, Jan 02, 2025 at 10:32:23PM -0500, Suresh Veliveli wrote: >>> > This is another instance where the replication stops. >>> > >>> > aaa-prod-aws-12:1636 >>> > # requesting: contextCSN >>> > contextCSN: *20250102015911.702871Z#000000#000#000000* >>> > >>> > *Master logs:* >>> > Jan 1 20:59:18 aaa-prod-master-1 slapd[3281130]: conn=1035 op=1 >>> > syncprov_sendresp: >>> > cookie=rid=152,csn=20250102015911.686467Z#000000#000#000000 >>> > Jan 1 20:59:18 aaa-prod-master-1 slapd[3281130]: conn=1035 op=1 >>> > syncprov_sendresp: >>> > cookie=rid=152,csn=20250102015911.702871Z#000000#000#000000 >>> > >>> > Nothing about rid=152 is logged after the above >>> >>> Hi Suresh, >>> you shouldn't be searching for the rid= on the provider, you might use >>> it to find the relevant "conn=xxx op=yyy" string and then search for >>> that. >>> >>> When you encounter this stall, could you do a 'thread apply all bt full' >>> on the provider? >>> >>> Given you also reported a crash in the server, where are you getting >>> packages from? Are you sure you are loading all modules from there and >>> not from an old version etc.? Would you be able to attach the provider >>> logs with at least sync+stats log level enabled? You can redact any >>> confidential information as needed. >>> >>> Thanks, >>> >>> -- >>> Ondřej Kuzník >>> Senior Software Engineer >>> Symas Corporation http://www.symas.com >>> Packaged, certified, and supported LDAP solutions powered by OpenLDAP >>> >> >> >> -- >> Suresh Veliveli >> Sr. UNIX Systems Engineer >> Georgetown University >> University Information Services | Security Infrastructure and >> Policy-Identity and Collaboration >> 202-262-6676 (cell) | 202-687-3108 (work) >> > > > -- > Suresh Veliveli > Sr. UNIX Systems Engineer > Georgetown University > University Information Services | Security Infrastructure and > Policy-Identity and Collaboration > 202-262-6676 (cell) | 202-687-3108 (work) > -- Suresh Veliveli Sr. UNIX Systems Engineer Georgetown University University Information Services | Security Infrastructure and Policy-Identity and Collaboration 202-262-6676 (cell) | 202-687-3108 (work)
GNU gdb (GDB) Rocky Linux 10.2-13.el9 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /var/services/openldap/libexec/slapd... [New LWP 192314] [New LWP 192300] [New LWP 192306] [New LWP 192304] [New LWP 192310] [New LWP 192311] [New LWP 192305] [New LWP 192313] [New LWP 192302] [New LWP 192315] [New LWP 192308] [New LWP 192301] [New LWP 192307] [New LWP 192318] [New LWP 192309] [New LWP 192316] [New LWP 192312] [New LWP 192317] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/var/services/openldap/libexec/slapd -h ldap://*:389 ldaps://*:636 -f /var/serv'. Program terminated with signal SIGSEGV, Segmentation fault. #0 connection_abandon (c=0x7f9eb4ad0078) at connection.c:714 [Current thread is 1 (Thread 0x7f85243fa640 (LWP 192314))] Thread 18 (Thread 0x7f85233f8640 (LWP 192317)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 17 (Thread 0x7f85253fc640 (LWP 192312)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 16 (Thread 0x7f8523bf9640 (LWP 192316)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 15 (Thread 0x7f8536ffd640 (LWP 192309)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 14 (Thread 0x7f8522bf7640 (LWP 192318)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 13 (Thread 0x7f8537fff640 (LWP 192307)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 12 (Thread 0x7f8553fff640 (LWP 192301)): #0 0x00007f9eccb0e21e in epoll_wait () from /lib64/libc.so.6 #1 0x00000000004421ee in slapd_daemon_task (ptr=0xaa8880) at daemon.c:2844 #2 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #3 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 11 (Thread 0x7f85377fe640 (LWP 192308)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 10 (Thread 0x7f8513fff640 (LWP 192315)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 9 (Thread 0x7f85537fe640 (LWP 192302)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 8 (Thread 0x7f8524bfb640 (LWP 192313)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 7 (Thread 0x7f85512fa640 (LWP 192305)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 6 (Thread 0x7f8525bfd640 (LWP 192311)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 5 (Thread 0x7f8534cf8640 (LWP 192310)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 4 (Thread 0x7f8551afb640 (LWP 192304)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 3 (Thread 0x7f8541bfd640 (LWP 192306)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca88fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 #2 0x00007f9ecd408425 in ldap_pvt_thread_cond_wait (cond=0xac80b8, mutex=0xac8090) at thr_posix.c:294 #3 0x00007f9ecd406b18 in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1041 #4 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #5 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 Thread 2 (Thread 0x7f9eccc87f00 (LWP 192300)): #0 0x00007f9ecca8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6 #1 0x00007f9ecca8b6d3 in __pthread_clockjoin_ex () from /lib64/libc.so.6 #2 0x00007f9ecd408366 in ldap_pvt_thread_join (thread=140210616661568, thread_return=0x0) at thr_posix.c:214 #3 0x0000000000443c2e in slapd_daemon () at daemon.c:3377 #4 0x000000000041c8f2 in main (argc=9, argv=0x7ffecb55daf8) at main.c:869 Thread 1 (Thread 0x7f85243fa640 (LWP 192314)): #0 connection_abandon (c=0x7f9eb4ad0078) at connection.c:714 #1 0x00000000004460d5 in connection_closing (c=0x7f9eb4ad0078, why=0x5db380 <conn_lost_str> "connection lost") at connection.c:785 #2 0x0000000000447d18 in connection_read (s=31, cri=0x7f85243f99a0) at connection.c:1453 #3 0x000000000044741b in connection_read_thread (ctx=0x7f85243f99f0, argv=0x1f) at connection.c:1260 #4 0x00007f9ecd406bed in ldap_int_thread_pool_wrapper (xpool=0xac8080) at tpool.c:1059 #5 0x00007f9ecca89c02 in start_thread () from /lib64/libc.so.6 #6 0x00007f9eccb0ec40 in clone3 () from /lib64/libc.so.6 No core file now.
