andygrove commented on issue #5355:
URL:
https://github.com/apache/datafusion-comet/issues/5355#issuecomment-5287080065
## Candidate C measured
Extended the existing `shuffle_block_schema_encoding` group with a `try_new`
case so the
per-batch construction cost can be compared directly against the per-batch
encode cost.
Apple silicon, `cargo bench -p datafusion-comet-shuffle --bench
shuffle_writer`:
| schema | `write_batch` | `try_new` | `try_new` as % of encode |
|---|---|---|---|
| flat (50 x Int32) | 74.27 µs | 4.16 µs | 5.6% |
| nested (4 cols, depth 6, 3-field leaf) | 35.70 µs | 5.87 µs | **16.4%** |
So the waste is real and it does skew nested, but it is more modest than I
implied when
filing: `try_new` only grows 4.16 -> 5.87 µs (1.4x) going from the flat to
the nested
schema here, and the nested case's percentage is high partly because its
`write_batch` is
*cheaper* (12 leaf arrays vs 50).
Two caveats on reading these numbers:
- This is the IPC-encode step in isolation. `process_sorted_row_partition`
also runs
`append_columns` per batch to convert `UnsafeRow` -> Arrow, which is very
likely the
dominant cost of that function. So Candidate C's share of the *whole*
columnar shuffle
write is smaller than 16%.
- The nested fixture is only 4 top-level columns. A schema with hundreds of
leaf fields
should scale `try_new` up further, since it walks `flattened_fields()` and
encodes the
whole flatbuffer — but that is extrapolation, not measured.
Conclusion: worth fixing (it is a one-line hoist that cannot regress
anything), but on
this evidence it is **not** on its own an explanation for "slower than
Spark". The scan
half (Candidate A) and the row-conversion cost in `append_columns` remain
the more likely
places for the reported gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
