freemandealer opened a new issue, #12359: URL: https://github.com/apache/doris/issues/12359
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? Load OOM with large exec_mem_limit and/or high load concurrency ### What You Expected? Load successfully. ### How to Reproduce? 32GB ram default mem_limit (50%) large exec_mem_limit, e.g. 8G high load job concurrency, e.g. 100 ### Anything Else? In OlapTableSink::send, before the batch actually sent by RPC, there is a pending queue to hold batchs. Each NodeChannel has its own queue, with size up to 1/20 exec_mem_limit. If exec_mem_limit set to 8GB, the queue could be maximum 400MB. Plus, when parallel load is enabled and multiple load jobs are issused, the pending queues from different NodeChannels all together consume a large amount of memory. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
