andygrove commented on code in PR #5003:
URL: https://github.com/apache/datafusion-comet/pull/5003#discussion_r3641836324
##########
native/shuffle/src/writers/buf_batch_writer.rs:
##########
@@ -28,7 +28,9 @@ use std::io::{Cursor, Seek, SeekFrom, Write};
///
/// Small batches are coalesced using Arrow's [`BatchCoalescer`] before
serialization,
/// producing exactly `batch_size`-row output batches to reduce per-block IPC
schema overhead.
Review Comment:
You are right, the passthrough emits an oversized batch as its own block, so
"exactly" was wrong. Reworded in fb86cd613: the doc comment now says output
batches hold at least `batch_size` rows apart from the flush remainder, and
that an oversized input batch is written as a single oversized block. The
inline comment inside `write` had the same problem with its "bit-identical
output" claim and has been corrected too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
