Doris-Breakwater commented on issue #66997: URL: https://github.com/apache/doris/issues/66997#issuecomment-5354634609
Breakwater-GitHub-Analysis-Slot: slot_c701f28ca940 Preliminary maintainer analysis: this is a real, high-severity process-thread exhaustion report, but the evidence does **not yet establish that one `RScan_normal` pool exceeded its limit or failed to shrink**. That interpretation conflicts with the 3.1.4 source path, so the root cause should remain unconfirmed until pool identity and user-space stacks are collected. ### Verified from the 3.1.4 code * `RScan_normal` is the remote scanner scheduler owned by the `normal` workload-group object. It is created once per workload-group object and backed by the generic Doris `ThreadPool` ([workload_group.cpp](https://github.com/apache/doris/blob/3.1.4-rc02/be/src/runtime/workload_group/workload_group.cpp#L494-L513)). * The pool only creates another worker while `num_threads + num_threads_pending_start < max_threads` ([threadpool.cpp](https://github.com/apache/doris/blob/3.1.4-rc02/be/src/util/threadpool.cpp#L434-L459)). Idle workers above `min_threads` exit after the pool's 500 ms idle timeout ([threadpool.cpp](https://github.com/apache/doris/blob/3.1.4-rc02/be/src/util/threadpool.cpp#L551-L579)); the 3.1.4 default remote-scan minimum is 8, not 512. * `thread_pool_active_threads=0` reports workers currently executing tasks; it is **not** the total number of live workers. 3.1.4 exposes `active_threads` and `max_threads` as metrics, but not `num_threads`. Therefore those two metric values alone cannot show whether the pool retained workers. * Each pool metric entity has a unique `id` label in addition to `thread_pool_name` and `workload_group` ([threadpool.cpp](https://github.com/apache/doris/blob/3.1.4-rc02/be/src/util/threadpool.cpp#L290-L320)). That label was omitted from the issue sample and is important here. * On Linux, a pthread created inside an `RScan_normal` worker may inherit the creator's `comm` name unless it explicitly renames itself. Consequently, `ps ... comm=RScan_normal` is strong correlation with the scan path but is not proof that every such TID is a worker tracked by this Doris pool. Thus, a single intact pool following this code cannot account for 20k tracked workers with `max_threads=512`. The two leading **unverified** explanations are (a) multiple live/stale pool instances with the same display name but different metric IDs, or (b) threads created below the scan worker (for example by an I/O/dependency path) that inherited `RScan_normal`. A third possibility—pool accounting corruption or a build differing from the published tag—needs evidence before being asserted. ### Released fix / PR assessment I could not verify any 3.1.x or 3.0.x release or backport that fixes this exact symptom. The inspected 3.0 and 3.1 code has the same per-pool cap and idle-shrink mechanism. I also reviewed [#65112](https://github.com/apache/doris/pull/65112); it releases workload-group scheduler-owned queues during BE shutdown and does not demonstrate a fix for periodic runtime creation of thousands of idle OS threads. It should not be presented as the fix for this issue. Accordingly, there is currently no evidence-based patch version or PR to recommend for this report. ### Information needed to prove the owner/root cause Please collect the following from one affected BE, ideally immediately before and after two scheduled runs: 1. The **complete raw** `thread_pool_active_threads` and `thread_pool_max_threads` metric lines for `RScan_normal`, retaining the `id` label, plus the number of distinct IDs. Do not aggregate away `id`. 2. BE log lines containing `[topic_publish_wg]update workload group finish` and `thread info=` over the same time window. In 3.1.4 these include `[r_scan num:<...>, real_num:<...>, min_num:<...>, max_num:<...>]`, which directly reports the pool's tracked worker count and configured minimum/maximum. 3. User-space backtraces (with symbols if available) from 5–10 representative `RScan_normal` TIDs: include both old TIDs that survived several jobs and TIDs created by the latest job. `/proc/.../stack`, `wchan=futex`, or syscall 202 only shows the kernel wait point and is insufficient; the user-space frames above the futex are decisive. 4. The exact BE build hash/image digest from startup logs, not only the `3.1.4` tag, and the values of `doris_max_remote_scanner_thread_pool_thread_num`, `doris_scanner_min_thread_pool_thread_num`, and `doris_scanner_thread_pool_thread_num`. 5. `normal` workload-group ID/version and its `max_remote_scan_thread_num` / `min_remote_scan_thread_num` properties, plus the query IDs and timestamps of two reproducing jobs. Also state the storage-vault credential mode (static AK/SK vs. role/default provider) without including any credentials. Interpretation of the first two items is straightforward: * Multiple metric IDs or workload-group IDs would support multiple pool instances/lifecycle leakage. * One metric ID with `r_scan num/real_num <= 512` while `/proc` shows 20k names would prove that most named TIDs are not workers tracked by that pool; their user-space stacks will identify the real owner. * One metric ID with `r_scan num/real_num > 512` would contradict the published accounting guard and make the exact build plus a core/backtrace essential. Until then, reducing job frequency/concurrency and restarting before approaching `pids.max` are reasonable containment measures. Raising `pids.max` only delays the abort. Lowering `max_remote_scan_thread_num` can be used as a controlled diagnostic—check whether the per-run TID delta scales with it—but should not yet be described as a fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
