andygrove opened a new issue, #7078: URL: https://github.com/apache/datafusion-comet/issues/7078
### What is the problem the feature request solves? The per-frame fixed cost of the native Celeborn writer is high, and it dominates when frames are small. In `native/shuffle/src/writers/rss/rss_partition_writer.rs` and `shuffle_block_writer.rs`, each frame: - takes two admission round trips (`reserve_partition_data` for planning scratch, released, then again for the encoding bound) and several JNI crossings to reserve, push, release and register the callback; - re-serializes the IPC schema message through `StreamWriter` (the remote writer uses `SchemaEncoding::Fallback`, so the pre-encoded schema used by local blocks is not available); - allocates a fresh output buffer and encoder, and releases the zstd context after every admitted encode; - walks every nested child up to four times to estimate sizes (`estimated_pre_compaction_ipc_data_size`, `estimated_compaction_scratch`, `estimated_minimum_compacted_ipc_data_size`, each calling `to_data()`), and rebases List and Map offsets on every split; - runs dictionary compaction unconditionally (`compact_dictionary_columns`); - uses a fixed 4 KiB metadata slack (`IPC_METADATA_RESERVATION_BYTES`) for the first output capacity, which a wide schema with codec None or incompressible data overflows, so the batch is dropped and encoded again. ### Describe the potential solution - Fold the planning reservation into the encoding reservation, and batch the JNI calls per frame. - Pre-encode the schema when the schema has no dictionaries (as the local writer does) and reuse it per frame. - Compute the size estimates in one walk and cache them for splits. - Skip dictionary compaction when no dictionary column is present or the dictionaries are already minimal. - Size the first output capacity from the schema (metadata size is known per schema) instead of a constant. Add an RSS-path micro-benchmark with small frames first, so each step shows up. ### Additional context Finding C3 in #5905. Related: #6654 (payload copies) and the sibling issues on reservation size and frame coalescing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
