tohuya6 opened a new pull request, #25800: URL: https://github.com/apache/datafusion/pull/25800
## Which issue does this PR close? - Closes #25791. ## Rationale for this change Sorting Utf8View data can fail with "Failed to allocate" under memory limits that the same data as Utf8 fits in. The sort's reservation charges a view data buffer every time it is referenced: repeated in one array, shared across columns, and shared across the sorted output chunks. ## What changes are included in this PR? - `get_sliced_size` counts each view data buffer once per batch. - `sort_batch_stream` charges the sorted chunks through `RecordBatchMemoryCounter`, the same approach as #22862. - `ReservationStream` releases memory with the same counter, so it frees what was reserved. - Removes the spill workspace loan from `sort_batch_stream`, along with `test_single_batch_spill_returns_live_workspace_loan_on_drop` from #24740. The loan only ran when the sorted output cost more than the input estimate, which was this over-count. The loan in the spill path is unchanged. ## What is the testing strategy for this PR? - `test_sort_charges_shared_output_buffers_once` sorts a Utf8View batch into 4 chunks under a memory limit equal to the input's reservation. It fails on main. - `sliced_size_counts_repeated_view_buffers_once` checks that a buffer listed 3 times in an array, across 2 columns, is counted once. It fails on main. - The existing sort, spill, stream and memory_limit tests pass. - Local repro (2M rows, Utf8View payload, 10 runs per limit): at 1g/2g/3g/4g/8g, main passes 0/1/0/1/1 and this PR passes 0/7/8/6/10. Compacting sparse view arrays, as suggested in the issue, is left for a follow-up. ## Are there any user-facing changes? Sorts over Utf8View data reserve less memory. No public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
