(doris) branch branch-2.1 updated: [fix](multi table) fix single stream multi table memory leak (#38255) (#38824)

[dataroaring]

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
     new 6035edad0b1 [fix](multi table) fix single stream multi table memory 
leak (#38255) (#38824)
6035edad0b1 is described below

commit 6035edad0b19a0cbe41ea97406651594d6a21d2c
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Aug 4 22:12:44 2024 +0800

    [fix](multi table) fix single stream multi table memory leak (#38255) 
(#38824)
    
    pick (#38255)
    
    We meet OOM when using single stream multi table
    
    
    
![image](https://github.com/user-attachments/assets/748e9914-d591-4f41-8b28-412d3cecc841)
    
    It exist memory leak, and heap profile like:
    
    
    
![image](https://github.com/user-attachments/assets/af30c593-88ea-44f6-bba1-82436b13f99f)
    
    The stream load context will not release in some exception conditions as
    plan failed for high concurrency causing timeout when obtaining read
    lock. It is introduced by https://github.com/apache/doris/pull/35458
    
    The solution effect is shown in the following figure, which can run
    stably with a small amount of memory
    
    
    
![image](https://github.com/user-attachments/assets/4483e0a5-6c0c-4cdc-b8ed-3408da6a86b2)
---
 be/src/io/fs/multi_table_pipe.cpp | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/be/src/io/fs/multi_table_pipe.cpp 
b/be/src/io/fs/multi_table_pipe.cpp
index fa38b6440c1..4469174211e 100644
--- a/be/src/io/fs/multi_table_pipe.cpp
+++ b/be/src/io/fs/multi_table_pipe.cpp
@@ -324,6 +324,19 @@ void MultiTablePipe::_handle_consumer_finished() {
     _ctx->number_filtered_rows = _number_filtered_rows;
     _ctx->number_unselected_rows = _number_unselected_rows;
     _ctx->commit_infos = _tablet_commit_infos;
+
+    // remove ctx to avoid memory leak.
+    for (const auto& pair : _planned_tables) {
+        if (pair.second) {
+            
doris::ExecEnv::GetInstance()->new_load_stream_mgr()->remove(pair.second->id);
+        }
+    }
+    for (const auto& pair : _unplanned_tables) {
+        if (pair.second) {
+            
doris::ExecEnv::GetInstance()->new_load_stream_mgr()->remove(pair.second->id);
+        }
+    }
+
     LOG(INFO) << "all plan for multi-table load complete. number_total_rows="
               << _ctx->number_total_rows << " number_loaded_rows=" << 
_ctx->number_loaded_rows
               << " number_filtered_rows=" << _ctx->number_filtered_rows


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org