felix199103 commented on issue #66997: URL: https://github.com/apache/doris/issues/66997#issuecomment-5355584622
Thanks for the clarification — we agree that `active_threads=0` does not mean OS thread count, and that `ps` `comm=RScan_normal` alone does not prove each TID is a ThreadPool worker. Here is the evidence we have so far. ### Version / build - Doris BE: `doris-3.1.4-rc02` / commit `7f5ba43de6c7f148e1dacdb7cb838394d2ec1537` - Base image: `apache/doris:be-3.1.4` (+ only a `paimon-s3` jar in custom_lib) - Deploy mode: cloud / compute group ### Workload group (`SHOW WORKLOAD GROUPS`) Only one row: | field | value | |---|---| | Id / Name | 1 / normal | | cpu_share | 1024 | | memory_limit | 30% | | enable_memory_overcommit | true | | max_concurrency | 2147483647 | | scan_thread_num | **-1** (default) | | max_remote_scan_thread_num | **-1** (default) | | min_remote_scan_thread_num | **-1** (default) | | spill low/high | 50% / 80% | | running_query_num / waiting_query_num | 0 / 0 | BE log effective values for the same WG: - `scan_thread_num=48` - `max_remote_scan_thread_num=512` - `min_remote_scan_thread_num=8` - TG list size = 1 ### ThreadPool metrics (single pool id) thread_pool_name="RScan_normal" workload_group="normal" id="cc4d753dbd47f8b5-167c63f4c3af5fb3" max_threads=512 active_threads=0 No second `RScan_*` pool id observed. ### BE r_scan accounting (repeated) [r_scan num:8, real_num:8, min_num:8, max_num:512] `real_num` stays at **8** even when OS has ~20k threads named `RScan_normal`. ### OS observation (why we originally suspected the pool) - Most threads are named `RScan_normal` and idle on futex - Creation pattern: batches of ~85–111 every ~10 minutes after BE start (not one giant burst) - Reducing a heavy ADS wide-table job from 1min → 10min clearly slowed growth - Eventually hits `pids.max` (~37776) → `Could not create thread (error 11)` → BE abort ### Current interpretation (aligned with your note) This looks more like case **(b)**: many OS threads inherit the name `RScan_normal`, but are **not** the tracked ThreadPool workers (`real_num=8`). We no longer claim “one RScan pool exceeded max_threads=512 / failed to shrink past 512”. ### Still missing (will follow up) Userspace backtraces (`bt`) for: - several **old** `RScan_normal` TIDs - several **newly created** TIDs from a fresh batch Please let us know if any other FE/BE dump would help before that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
