andygrove opened a new pull request, #5004: URL: https://github.com/apache/datafusion-comet/pull/5004
## Which issue does this PR close? Addresses the "The single-partition path coalesces twice" item in #5002. That issue is an umbrella tracking several shuffle-writer optimizations, so this PR does not close it. ## Rationale for this change `SinglePartitionShufflePartitioner` buffered small batches and `concat_batches`'d them to `batch_size` before handing the result to the long-lived `BufBatchWriter`. That writer's own `BatchCoalescer` then copies all the rows again into `batch_size`-row IPC blocks. The concat layer predates the coalescer in the writer and is now redundant: the coalescer already produces exactly `batch_size`-row blocks regardless of how the input is chunked, so the partitioner's concatenation is pure extra work that copies every row an additional time. ## What changes are included in this PR? `SinglePartitionShufflePartitioner` now streams every incoming batch straight to the `PartitionWriter` and lets the writer's `BatchCoalescer` do the batching. The internal `buffered_batches` / `num_buffered_rows` state, the `add_buffered_batch` / `concat_buffered_batches` helpers, and the now-unused `batch_size` field are removed. Because the coalescer's output is a pure function of the concatenated row stream and `batch_size` (independent of input chunking), the IPC block boundaries and bytes are unchanged. This removes one full copy of the shuffle payload in single-partition shuffles. ## How are these changes tested? Existing `datafusion-comet-shuffle` unit tests, including `test_single_partition_shuffle_writer`, continue to pass. A new `single_partition_coalesces_to_batch_size_in_order` test drives the partitioner directly with a mix of sub-`batch_size` batches and a larger-than-`batch_size` batch, then reads the output data file back and asserts the blocks are coalesced to `batch_size` (plus a tail) and that every row roundtrips in order. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
