jayzhan211 commented on issue #26711: URL: https://github.com/apache/datafusion/issues/26711#issuecomment-6554324101
I tried the shared spill and merge part of this EPIC as two PRs. **#26987** bounds the size in bytes of every batch in a sorted spill run, for the sort and both aggregate streams, and caps the merge's output at the same bound. With that, the merge no longer needs to re-spill runs with halved batches, so it removes `split_spill_file_in_half`, the per-run batch-size limit and the single-row scan. It fixes the reproducers in #25851 and #25910. It reuses the batch splitter from #25910 and the output cap approach from #25853, so it would replace both. **#26984**, stacked on #26987, replaces the aggregate-only `with_replay_headroom` and `with_intermediate_merge_sizing`, and the `with_fan_in_headroom` that #26264 proposes, with one admission rule for every caller: the formula from #26264. With nothing else using memory, it adds merge passes to a spilled sort (numbers are in the PR), so it needs `sort_tpch` results before it can be judged. Two sub-issues here look done already: - #16908: `max_spill_merge_fan_in` from #23066 is the proposed cap on merge degree. - #18261: view GC now happens inside `InProgressSpillFile`, since #21633. @adriangb @andygrove @MisterRaindrop, these overlap your PRs. Would you rather review #26987 and #26984 in place of yours, or land yours first and have me rebase on top? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
