andygrove opened a new pull request, #4934: URL: https://github.com/apache/datafusion-comet/pull/4934
## Which issue does this PR close? N/A ## Rationale for this change Optimize existing expression. ## What changes are included in this PR? `array_compact` only removes null elements. When the values buffer contains no null elements there is nothing to remove, so the output is bit-identical to the input. This adds a fast path to `compact_list` that returns the input array unchanged in that case, reusing its buffers zero-copy and skipping the per-element `MutableArrayData::extend` loop. The null-containing path is left byte-for-byte unchanged, so arrays with sparse or dense element nulls take exactly the same code as before and do not regress. This deliberately avoids the per-row run-batching approach that regressed the dense-null shape in a previous attempt (#4886). Also adds Rust unit tests (the expression previously had only SQL file / Scala coverage) and a criterion benchmark covering no-null, null-row, sparse-null, and dense-null shapes. ## How are these changes tested? New Rust unit tests assert the fast path is bit-identical to the input (including null and empty rows) and that the null-removal path is unchanged; existing SQL file tests and `CometArrayExpressionSuite` continue to cover end-to-end behavior. Benchmark (criterion), baseline `main` vs this branch, 8192-row list columns: ``` array_compact: no nulls short: 316 µs -> 287 ns (~99.9% faster) array_compact: no nulls long: 2.46 ms -> 254 ns (~99.99% faster) array_compact: no element nulls, null rows: 296 µs -> 258 ns (~99.9% faster) array_compact: sparse nulls: 489 µs -> 488 µs (within noise) array_compact: dense nulls: 258 µs -> 257 µs (within noise) ``` The null-free shapes hit the zero-copy fast path; the sparse/dense-null shapes run the unchanged loop and are within noise across repeated samples. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
