Weijun-H opened a new pull request, #25406: URL: https://github.com/apache/datafusion/pull/25406
## Which issue does this PR close? - Closes #25405. ## Rationale for this change External sort can retain already-consumed input batches after producing a partial output batch. This keeps memory reserved longer than needed and can increase memory pressure for sorts that emit output in multiple chunks. ## What changes are included in this PR? This PR updates `BatchBuilder` cleanup after output emission to retain only batches that are still referenced by pending row indices or live cursors. It remaps remaining indices and cursors after dropping consumed batches. The change is limited to earlier release of consumed input batches. It does not add sort byte-targeting, spill admission changes, or new configuration. ## What is the testing strategy for this PR? Added a `BatchBuilder` regression test that emits a partial output batch and checks both directions: - the consumed input batch is released and its memory reservation is returned; - batches still needed by pending indices or a live cursor remain available and produce the expected later output. Also ran: - `cargo fmt --check` - `cargo test -p datafusion-physical-plan sorts::builder --lib` - `git diff --check HEAD^ HEAD` ## Are there any user-facing changes? No API or configuration changes. This only releases internal sort input batch memory earlier. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
