peterxcli opened a new pull request, #5804: URL: https://github.com/apache/datafusion-comet/pull/5804
## Which issue does this PR close? Closes #5793. ## Rationale for this change `shuffle_bench` built its input with a default `SessionConfig`, and DataFusion defaults `schema_force_view_types` to true, so string and binary columns reached `ShuffleWriterExec` as `Utf8View` and `BinaryView`. Comet does not hand the shuffle writer those types: the serde maps Spark `String` to `Utf8`, and the planner casts UDF results back from the view variants with the comment that Comet does not yet support them. The benchmark was measuring the writer on a data shape production does not produce. View arrays are also the input this writer handles worst. `BufBatchWriter` configures its `BatchCoalescer` with `biggest_coalesce_batch_size` of `batch_size - 1`, so a batch at or above `batch_size` rows bypasses the coalescer and is serialized verbatim, and an interleaved view array still references the backing data buffers of every input batch it drew rows from. Measured on a 5 column input with 2 string columns, 4M rows and 200 partitions, comparing the output file against the writer's own `data size` metric: | batch size | output file | data size | ratio | | --- | --- | --- | --- | | 2048 | 33.17 GiB | 236.95 MiB | ~143x | | 4096 | 14.86 GiB | 242.07 MiB | ~63x | | 8192 | 7.62 GiB | 242.33 MiB | ~32x | | 16384 | 4.00 GiB | 243.96 MiB | ~17x | | 20480 | 294.92 MiB | 245.45 MiB | ~1.2x | The cliff at 20480 is where a partition's 20000 rows stop exceeding the threshold and go through the coalescer, which compacts the views. Any string-heavy number this benchmark produced was dominated by amplification real Comet does not have, with encode time and write time inflated in proportion. #5198 points at this benchmark for measuring its items 1 and 6. ## What changes are included in this PR? - Disable `datafusion.execution.parquet.schema_force_view_types` in the benchmark's `SessionConfig` so the input matches what Comet's planner produces. - Check the schema that actually reaches the writer. The header schema is read separately through `ParquetRecordBatchReaderBuilder`, so it reported the file's `Utf8` rather than the `Utf8View` that executed and the mismatch was invisible in the output. `reject_view_types` fails the run instead, recursing through list, struct, map and dictionary children. - Name `Utf8View` and `BinaryView` in `describe_schema` rather than letting them fall through to "other". ## How are these changes tested? Six unit tests on `reject_view_types` cover a plain schema, top-level `Utf8View` and `BinaryView`, and views nested behind a list, a struct and a dictionary. Verified end to end by re-running the sweep above. Output is now flat at ~238 MiB across every batch size from 2048 to 20480, which is the expected behaviour since output size should not depend on batch size, and encode time at batch size 2048 drops from 7.580s to 0.019s. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
