andygrove opened a new issue, #7077: URL: https://github.com/apache/datafusion-comet/issues/7077
### What is the problem the feature request solves? The native Celeborn writer emits one frame per partition chunk per spill, with no coalescing, so at large partition counts it sends many small frames. Each frame pays the full per-frame cost (see the sibling issue on per-frame crossings and admission). `RssPartitionWriter::push_batches` encodes and pushes every batch it is given for a partition immediately, and `CelebornShufflePartitionPusher` calls the Celeborn client's `pushOrMergeData` with `doPush=true` and `shouldMerge=true`, which appears to push each frame at once instead of letting the client merge small pushes (re-check against the Celeborn source when picking this up). The local writer coalesces small batches through `BatchCoalescer`; the remote path has no equivalent. ### Describe the potential solution Stage small per-partition batches natively across write calls and encode them only when they reach the target batch size (or at spill and finish). Alternatively pack several complete blocks for the same partition into one push; the reader already handles multiple blocks per fetched stream. Measure with the writer bench's high-partition cases on the RSS path (`rss_execution_tests.rs` has the harness) and report frames pushed and bytes per frame. ### Additional context Finding C2 in #5905. Related: #6654. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
