Jason Fehr has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/24159 )
Change subject: [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing ...................................................................... [WIP] IMPALA-14863: Fixes Potential OOM During Row Batch Processing Impala checks at the end of every row batch to determine if the query or process memory limit has been exceeded. If the batch processing requires evaluating expressions that allocate memory for their results (such as UPPER() or LOWER()), that memory is allocated without checking any memory limits. If a large number of these expression evaluations are performed, memory will be consume until the process memory limit is reached. At that point, the executor begins failing other queries with cannot allocate memory errors as they request memory. However, the row batch processing continues to run since it does no memory limit checking during batch processing. This patch determines the proper soft memory limit based on the lesser of the query and process soft limits. When expression evaluation results memory pool exceeds that threshold, the memory pool is freed during which row batch processing is paused. Generated-by: Github Copilot (GPT-5.3-Codex) Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af --- M .gitignore M be/src/codegen/gen_ir_descriptions.py M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exprs/scalar-expr-evaluator.cc M be/src/exprs/scalar-expr-evaluator.h M be/src/runtime/mem-tracker-test.cc M be/src/runtime/mem-tracker.cc 9 files changed, 144 insertions(+), 50 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24159/4 -- To view, visit http://gerrit.cloudera.org:8080/24159 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic54b5c39e1388681275681f22e61b27728dba5af Gerrit-Change-Number: 24159 Gerrit-PatchSet: 4 Gerrit-Owner: Jason Fehr <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
