Han-lai opened a new issue, #43550: URL: https://github.com/apache/doris/issues/43550
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version **Doris Version:** 2.1.5 **Stream Load Settings:** streaming_load_max_m*b = 10240 (10 GB) string_type_length_soft_limit_bytes = 2147483643 (limit for string type length) doris.batch.size = 1024 **Workload Group Settings:** ALTER WORKLOAD GROUP normal PROPERTIES("enable_memory_overcommit" = "false"); ALTER WORKLOAD GROUP normal PROPERTIES("memory_limit" = "70%"); ### What's Wrong? Execute the following SQL query on Doris `SELECT TimeStamp, UID, LineName, Messagee, source, Target, RID, Message FROM PK_MessageBody_data WHERE LineName='Line_02' AND MessageName='Production_A' AND toString(TimeStamp) > '2024-10-13 00:00:00.000 +0000' AND toString(TimeStamp) < '2024-10-14 00:00:00.000 +0000' AND length(MessageBody) >= 5000000 LIMIT 400; ` The Doris process throws the following error, indicating memory overcommit and cancellation:The Doris process throws the following error, indicating memory overcommit and cancellation: `ErrorCode:[Doris-01], ErrorDescription:[stream load error] - stream load error: [CANCELLED]GC wg for hard limit, wg id:1, name:normal, used:8.00 GB, limit:7.56 GB, backend:1x.xxx.xxx.xxx. cancel top memory used tracker <Load#Id=6c4423150a0475be-b705cd052d5d8bb2> consumption 8.00 GB. details:process memory used 4.12 GB exceed soft limit 9.72 GB or sys available memory 8.09 GB less than warning water mark 1.20 GB., Execute again after enough memory, details see be.INFO., see more in null. ` ### What You Expected? The query should run successfully without causing a memory overcommit error or canceling the load process, even when processing large message body data. ### How to Reproduce? Expected Behavior: The query should run successfully without causing a memory overcommit error or canceling the load process, even when processing large message body data. Actual Behavior: The load is canceled due to exceeding the memory limit (8.00 GB used vs. 7.56 GB limit). The process is stopped because the memory usage exceeds both the soft limit and available system memory. ### Anything Else? We have confirmed that the data being processed does not exceed the configured GC settings, and the memory usage aligns with the configured limits. The issue persists even though the data volume does not exceed the configured memory limits. The Doris configuration for memory management appears to be working correctly, but the memory overcommit error still occurs. Suggest reviewing the memory management and load handling mechanism to prevent load cancellations due to memory overcommit. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
