matanper opened a new issue, #68974: URL: https://github.com/apache/doris/issues/68974
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode, arm64). master is the same: - `be/src/common/daemon.cpp:322-324`: when `process_memory_usage() > mem_limit()`, the memory GC calls `MemoryReclamation::revoke_process_memory`. - `be/src/runtime/memory/memory_reclamation.cpp:225-237`: after queries, that cancels the largest loads until `process_full_gc_size` is freed (`process_full_gc_size = "10%"`, `config.cpp:182`). - `be/src/load/memtable/memtable_memory_limiter.cpp:199` and `:248`: the load limiter returns, or stops waiting, as soon as memtable memory is at or below `_load_safe_mem_permit` (5% of the 50% load limit, i.e. 2.5% of `mem_limit`), whatever the process memory is. - `memtable_memory_limiter.cpp:143-188`: the per-table gate compares a flat count of pending memtables with `table_memtable_flush_pending_count_limit` (default 10), and counts every load into the table. ### Description With many concurrent stream loads, a BE reaches its process memory limit, and the memory GC then cancels loads ("Cancel Top Memory task"). We saw hundreds of cancellations per BE per 10 minutes. Each cancelled load (~475 MB in flight in our case) is retried by the client from the start, so cancelling makes the overload worse. Reading the code, nothing throttles loads before that point: - **The GC can only cancel.** It frees 10% of the limit at a time, by cancelling the largest loads. It counts a cancelled task's memory as freed only for `revoke_memory_max_tolerance_ms` (3 s). Loads release memory more slowly than that, so one GC round tends to cancel more loads than it needs to. - **`MemTableMemoryLimiter` cannot see most of the pressure.** It waits while process memory is above the soft limit, but only while memtable memory is above 2.5% of `mem_limit`. Memory that loads hold outside memtables (flush, segment and index writers, S3 upload buffers, the stream load pipe and scanner) does not keep it waiting. So process memory can climb from the soft limit to the hard limit while load writes go on. - **The per-table gate counts memtables, not bytes.** Raising `table_memtable_flush_pending_count_limit` to keep loads moving lets queued memtables push the process into the GC. - **The wait has to be scoped to the writing load.** We first tried gating writes on table-wide pending flushes. With ~160 concurrent loads into one table, the table almost always has a flush pending, so every write waited until memory dropped or the wait timed out. It waited inside the shared send-batch pool, and that starved the other loads' batches and their EOS. In an off-CPU profile, send-batch threads spent 76.5 s per 2 s window in `handle_table_memtable_backpressure`, and loads waited 8–14 s at close while the CPU was idle. The existing count gate has the same scope: it is per table, not per load. ### Solution We have a patch. Every part is behind a mutable config and off by default. 1. Above `load_memory_backpressure_process_mem_percent` of `mem_limit`, set below the GC threshold, the limiter flushes memtables and makes load writes wait. This depends on process memory, not only on memtable memory. 2. The per-table gate also waits on a byte budget for queued plus flushing memtables (`load_memtable_pending_mem_limit_percent`), not only on the count. 3. The memory part of the gate waits only while the writer's own load has flushes pending (`load_memory_backpressure_wait_on_own_load`). The table-wide count limit is unchanged. 4. Memory waits are bounded (`load_memory_backpressure_max_wait_ms`) and honour load cancel. 5. Process GC frees only the overshoot plus a small margin (`load_memory_gc_free_extra_percent`), not a fixed 10%. Cancellation stays as the safety valve. ### How to Reproduce? Run many concurrent stream loads (hundreds per BE) into one table, with enough data per load that BE process memory reaches `mem_limit`. Then watch the `[MemoryGC]` log lines and the load failures with `MEM_LIMIT_EXCEEDED` / "Cancel Top Memory task". The `mm_limiter_*` bvars show that the memtable limiter is not waiting while this happens. ### Anything Else? The cancellation rate and the close-wait profile are production measurements. The claim that the GC over-cancels comes from reading the code. The patch has unit tests (`LoadMemoryBackpressureTest`), but we have not yet completed a production A/B with it. Letting more writes through above the threshold relies on the GC as the backstop, so `MEM_LIMIT_EXCEEDED` should be watched when tuning. Related: #68970 (upload buffers are part of the memory that this limiter does not see). ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
