zclllyybb commented on issue #65428: URL: https://github.com/apache/doris/issues/65428#issuecomment-4925793879
Breakwater-GitHub-Analysis-Slot: slot_9fe234b265b1 This content is generated by AI for reference only. Initial analysis from current `upstream/master` code inspection: This looks like a valid startup deadlock risk, not just a documentation mismatch. In current master, `be/src/common/phdr_cache.cpp` still exports a process-wide `dl_iterate_phdr` interposer. When the thread-local `use_phdr_cache` flag is false, that interposer calls `getOriginalDLIteratePHDR()`, and that function resolves the real loader function with `dlsym(RTLD_NEXT, "dl_iterate_phdr")` on the call path. The supplied stack is consistent with why this is unsafe: jemalloc profiling is already inside `malloc_init_hard()` and holds the jemalloc init mutex, then unwinding reaches `dl_iterate_phdr`, enters Doris' interposer, and `dlsym()` performs an allocation that re-enters jemalloc initialization. There is one important build-path detail to verify. Current master's intended Linux path is that jemalloc heap profiling uses GNU libunwind, and Doris' patched libunwind calls the narrower `doris_unwind_iterate_phdr(callback, data, ip)` hook instead of ordinary `dl_iterate_phdr`. The third-party build script also configures jemalloc with `--enable-prof-libunwind --disable-prof-libgcc` and refuses a build where `prof-libunwind` is not enabled or `prof-libgcc` is enabled. However, the reported stack contains `_Unwind_Backtrace` / `_Unwind_Find_FDE` frames, which is exactly the kind of libgcc-style fallback current master is trying to prevent. So the immediate cause may be either: 1. the BE was built or run with stale/mismatched third-party jemalloc artifacts where heap profiling still uses the libgcc `_Unwind_Backtrace` path, or 2. another early unwind path is still reaching the process-wide `dl_iterate_phdr` interposer before the original function has been resolved in a safe context. Either way, the code-level weak point is clear: the default branch of Doris' `dl_iterate_phdr` interposer should not perform lazy `dlsym()` while the caller may be in allocator initialization or unwinding. A robust fix should either avoid process-wide interposition for normal callers and keep the PHDR snapshot only behind `doris_unwind_iterate_phdr` / the minimal scoped stack-trace path, or ensure the original `dl_iterate_phdr` function pointer is obtained through a path that is proven not to allocate or re-enter jemalloc from inside the interposed function. Missing information needed to pin this down: - exact Doris commit SHA and whether third-party artifacts were rebuilt after the PHDR/libunwind changes in PR #64093; - jemalloc build evidence from `config.log` or the third-party build log showing `prof-libunwind: 1` and `prof-libgcc: 0`; - the startup environment that enabled profiling, especially `MALLOC_CONF` / `JE_MALLOC_CONF` and the BE launch command; - glibc and JDK versions, since the observed allocation is inside `dlsym()` while `libjvm.so` static initialization is running; - if available, symbol evidence from the affected BE binary showing whether jemalloc is using `unw_backtrace` or `_Unwind_Backtrace`. Recommended next steps for maintainers: - First verify the reporter's jemalloc profiling backend. If it is libgcc-backed, rebuild third-party artifacts from current master and confirm whether the startup hang disappears. - Independently fix the interposer hazard: do not call `dlsym(RTLD_NEXT, "dl_iterate_phdr")` lazily from the normal `dl_iterate_phdr` forwarding path. - Add a regression/manual startup test covering BE startup with jemalloc profiling enabled and JVM loading, because this failure happens before the service becomes ready and ordinary runtime tests may not exercise it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
