andygrove opened a new issue, #7080: URL: https://github.com/apache/datafusion-comet/issues/7080
### What is the problem the feature request solves? The shuffle benches in `native/shuffle/benches` do not cover several shapes that the performance review in #5905 flagged, so changes to those paths cannot show a win or catch a regression. What exists today: `shuffle_writer.rs` (codecs None/LZ4/Snappy/Zstd, a high-partition group, partitioning placement, schema encoding, IPC context), `shuffle_reader.rs` (codecs none and lz4, dictionary blocks, schema parse) and `row_columnar.rs` (primitive, struct, list and map conversion). ### Describe the potential solution Add the missing cases, each as its own small PR: - **Writer:** end-to-end wide schema (100+ columns); List, Struct and Map through `interleave`; a spill run under a memory limit; a compressed high-partition case (the existing high-partition group uses codec None). - **Reader:** zstd and snappy blocks; List<Struct> and Map blocks; a cached-schema variant against `decode_block_uncached` for nested schemas. - **Row to Arrow (`row_columnar.rs`):** strings (plain and dictionary-encoded), List<Struct>, Map<Utf8, _> and nullable data. The existing list case `list_conversion/elements_100` has data-dependent output sizes that #6855 asks someone to look at first. Record results in the PR description with `cargo bench -- --save-baseline`. ### Additional context "Benchmark coverage to add" section of #5905. Good first issue; the individual items are independent. Related: #6856, #6855. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
