weilai201 opened a new issue, #31637: URL: https://github.com/apache/doris/issues/31637
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 2.0.4-rc6 ### What's Wrong? 我现在有一台4和16GB内存的机器，上面部署了doris fe和be。be做了如下配置： mem_limit = 50% segment_cache_capacity=10000 disable_mem_pools=true load_process_max_memory_limit_percent=40 在使用datax持续写入数据过程中，发现be内存一直在增加，直到系统datax导入数据失败。 错误消息如下 `{"BeginTxnTimeMs":0,"Comment":"","CommitAndPublishTimeMs":0,"Label":"my_prefix291e78fc-ad72-4fdb-9d59-c28a8d04519a_45","LoadBytes":8376020,"LoadTimeMs":3146,"Message":"[INTERNAL_ERROR]cancelled: Process has no memory available, cancel top memory used load: load memory tracker <Load#Id=9f448e1aefe9d246-9a7f4d671b5aeeb6> consumption 11.09 MB, backend 192.168.1.103 process memory used 7.74 GB exceed limit 7.69 GB or sys available memory 1.46 GB less than low water mark 1.54 GB. Execute again after enough memory, details see be.INFO.\n\n\t0# doris::io::StreamLoadPipe::_append(std::shared_ptr<doris::ByteBuffer> const&, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0\n\t1# doris::io::StreamLoadPipe::append(std::shared_ptr<doris::ByteBuffer> const&) at /home/zcp/repo_center/doris_release/doris/be/src/io/fs/stream_load_pipe.cpp:162\n\t2# doris::StreamLoadAction::on_chunk_data(doris::HttpRequest*) at /home/zcp/repo_center/doris_release/doris/be/src/ common/status.h:442\n\t3# ?\n\t4# bufferevent_run_readcb_\n\t5# ?\n\t6# ?\n\t7# ?\n\t8# ?\n\t9# std::_Function_handler<void (), doris::EvHttpServer::start()::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/atomicity.h:98\n\t10# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0\n\t11# doris::Thread::supervise_thread(void*) at /var/local/ldb_toolchain/bin/../usr/include/pthread.h:562\n\t12# start_thread\n\t13# clone\n","NumberFilteredRows":0,"NumberLoadedRows":0,"NumberTotalRows":0,"NumberUnselectedRows":0,"ReadDataTimeMs":17,"Status":"Fail","StreamLoadPutTimeMs":3,"TwoPhaseCommit":"false","TxnId":2124,"WriteDataTimeMs":1918} ` 内存消耗截图： <img width="1837" alt="image" src="https://github.com/apache/doris/assets/12964080/e0b9d2a5-4e31-463a-bbb3-52f446cad88c";> ### What You Expected? be可以自动做内存释放！当内存不足时，可降低处理速度，提升系统稳定性 ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
