Userwhite opened a new issue, #65428: URL: https://github.com/apache/doris/issues/65428
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? Doris BE may hang during startup before the service becomes ready. The observed hang happens very early during process initialization. GDB shows only one thread, blocked in jemalloc initialization. The stack indicates a recursive jemalloc initialization path: 1. `libjvm.so` static initialization performs allocation. 2. jemalloc enters `malloc_init_hard()` and holds jemalloc init mutex. 3. jemalloc profiling initialization calls `_Unwind_Backtrace()`. 4. libunwind calls `dl_iterate_phdr()`. 5. Doris' PHDRCache interposes the process-wide `dl_iterate_phdr`. 6. Because `use_phdr_cache == false`, the interposer calls `getOriginalDLIteratePHDR()`. 7. `getOriginalDLIteratePHDR()` calls `dlsym(RTLD_NEXT, "dl_iterate_phdr")`. 8. `dlsym()` internally allocates memory and re-enters jemalloc initialization while the jemalloc init mutex is already held. This causes a startup deadlock. Relevant Doris code: ```cpp // be/src/common/phdr_cache.cpp DLIterateFunction getOriginalDLIteratePHDR() { void* func = dlsym(RTLD_NEXT, "dl_iterate_phdr"); if (!func) { throw std::runtime_error("Cannot find dl_iterate_phdr function with dlsym"); } return reinterpret_cast<DLIterateFunction>(func); } extern "C" int dl_iterate_phdr( int (*callback)(dl_phdr_info* info, size_t size, void* data), void* data) { if (!use_phdr_cache) { return getOriginalDLIteratePHDR()(callback, data); } return iteratePHDRCache(callback, data, 0); } This seems inconsistent with the comment in be/src/common/phdr_cache.h, which says normal code paths should keep using the original glibc dl_iterate_phdr, while Doris-patched libunwind should use the narrower doris_unwind_iterate_phdr hook. ``` Full Stack ``` #0 0x00007f2a73ff029c in __lll_lock_wait () from /lib/x86_64-linux-gnu/ libpthread.so.0 #1 0x00007f2a73fe9714 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/ libpthread.so.0 #2 0x000055b05f66f0c5 in malloc_mutex_lock_final (mutex=0x55b064f4bb50 <init_lock>) at ../include/jemalloc/internal/mutex.h:151 #3 je_malloc_mutex_lock_slow (mutex=0x55b064f4bb50 <init_lock>) at ../src/ mutex.c:90 #4 0x000055b05f639140 in malloc_mutex_lock (tsdn=0x0, mutex=<optimized out>) at ../include/jemalloc/internal/mutex.h:217 #5 malloc_init_hard () at ../src/jemalloc.c:2118 #6 0x000055b05f629cd7 in malloc_init () at ../src/jemalloc.c:298 #7 imalloc_init_check (sopts=<optimized out>, dopts=<optimized out>) at ../src/ jemalloc.c:2658 #8 imalloc (sopts=<optimized out>, dopts=<optimized out>) at ../src/ jemalloc.c:2689 #9 jecalloc (num=1, size=32) at ../src/jemalloc.c:2852 #10 0x00007f2a74001a45 in ?? () from /lib/x86_64-linux-gnu/libdl.so.2 #11 0x00007f2a7400140f in dlsym () from /lib/x86_64-linux-gnu/libdl.so.2 #12 0x000055b057940afc in (anonymous namespace)::getOriginalDLIteratePHDR () at ./be/build_Release/./be/src/common/phdr_cache.cpp:86 #13 dl_iterate_phdr (callback=0x55b063fe9420 <_Unwind_IteratePhdrCallback>, data=0x7ffdc7c0cd50) at ./be/build_Release/./be/src/common/phdr_cache.cpp:238 #14 0x000055b063febebe in _Unwind_Find_FDE () #15 0x000055b063fe7ab7 in uw_frame_state_for () #16 0x000055b063fe810d in uw_init_context_1 () #17 0x000055b063fe920c in _Unwind_Backtrace () #18 0x000055b05f673888 in je_prof_boot2 (tsd=tsd@entry=0x7f2a73e087c0, base=<optimized out>) at ../src/prof.c:693 #19 0x000055b05f639591 in malloc_init_hard () at ../src/jemalloc.c:2151 #20 0x000055b05f62718f in malloc_init () at ../src/jemalloc.c:298 #21 imalloc_init_check (sopts=<optimized out>, dopts=<optimized out>) at ../src/ jemalloc.c:2658 #22 imalloc (sopts=<optimized out>, dopts=<optimized out>) at ../src/ jemalloc.c:2689 #23 je_malloc_default (size=40) at ../src/jemalloc.c:2722 #24 0x00007f2a74d8e7a5 in NMTPreInitAllocation::do_alloc(unsigned long) () from /opt/tiger/jdk/jdk17/lib/server/libjvm.so #25 0x00007f2a74dba390 in os::malloc(unsigned long, MEMFLAGS, NativeCallStack const&) () from /opt/tiger/jdk/jdk17/lib/server/libjvm.so #26 0x00007f2a7457409a in AllocateHeap(unsigned long, MEMFLAGS, AllocFailStrategy::AllocFailEnum) () from /opt/tiger/jdk/jdk17/lib/server/libjvm.so #27 0x00007f2a74c47876 in LogOutputList::add_output(LogOutput*, LogLevel::type) () from /opt/tiger/jdk/jdk17/lib/server/libjvm.so #28 0x00007f2a74400a3f in _GLOBAL__sub_I_abstractDisassembler.cpp () from /opt/tiger/jdk/jdk17/lib/server/libjvm.so #29 0x00007f2a754f838a in ?? () from /lib64/ld-linux-x86-64.so.2 #30 0x00007f2a754f8486 in ?? () from /lib64/ld-linux-x86-64.so.2 #31 0x00007f2a754ea0ca in ?? () from /lib64/ld-linux-x86-64.so.2 #32 0x0000000000000001 in ?? () #33 0x00007ffdc7c0e8ea in ?? () #34 0x0000000000000000 in ?? () ``` ### What You Expected? BE should not deadlock during startup. ### How to Reproduce? jemalloc profiling is enabled ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
