andygrove opened a new pull request, #6856: URL: https://github.com/apache/datafusion-comet/pull/6856
## Which issue does this PR close? Part of #6855. ## Rationale for this change The C2R and R2C microbenchmarks covered nested types thinly: nested test data never contained nulls, collections held 2-5 simple elements, the isolated C2R bench and the native `row_columnar` bench had no strings or collections of structs, and nothing measured `CometSparkToColumnarExec` on nested input. Extending them found several cases where Comet is slower than Spark (filed under #6855). ## What changes are included in this PR? Benchmarks only, no production code. - `CometColumnarToRowBenchmark`: three new groups. Nullable nested types (null structs, arrays and maps, empty arrays, null elements), nested collections (array of struct, array of array, struct of array of struct, map of struct, decimals, timestamps and binary inside nested types), and large collections (100-element arrays, 20-entry maps). - `CometC2RIsolatedBench`: a nested schema (struct, array, map, array of struct, with nulls) alongside the flat one, at batch sizes 8192, 512 and 32. - `native/shuffle/benches/row_columnar.rs`: a small UnsafeRow encoder and 12 new cases covering `list<utf8>`, `list<struct>`, `list<list>`, lists with null elements, structs with strings, nested structs with strings, a struct holding a list, and `map<utf8,*>` variants. - `CometArrowWriterBenchmark`: nine more nested types (nested structs, `array<struct>`, `array<array>`, `struct<array>`, maps with array values, and others). - `CometSparkToColumnarBenchmark` (new): Spark Parquet scan converted to Arrow versus Spark and the native scan, for eight nested shapes, with plan checks on each arm. - `CometShuffleBenchmark`: ten nested shapes across Spark shuffle, Comet Spark shuffle, and JVM and native shuffle, plus a `--nested-only` flag. ## How are these changes tested? The new benchmarks were built in release mode and run on a Linux machine; results and findings are in #6855. The native bench encoder was checked to produce byte-identical rows to the existing hand-built `list` and `map` rows. `cargo fmt`, `cargo clippy --benches -D warnings`, `spotless` and `dev/ci/check-benchmark-runner.py` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
