andygrove opened a new pull request, #23481: URL: https://github.com/apache/datafusion/pull/23481
## Which issue does this PR close? - N/A (small performance improvement) ## Rationale for this change The Spark `slice` function's `calculate_start_end` helper reads the length of each list element in a per-row hot loop via `values.value(row).len()`. `GenericListArray::value(row)` materializes a new `ArrayRef` for the sublist on every iteration purely to call `.len()` on it. The length is already available from the offset buffer, so this allocation is pure overhead. Replacing it with `values.value_length(row)` reads the length directly from the offsets with no allocation. Benchmarked with the existing `datafusion/spark/benches/slice.rs` over 1M rows: | case | before | after | improvement | |------|--------|-------|-------------| | List(Int64), array args | 54.0 ms | 40.0 ms | ~26% faster | | List(Int64), scalar args | 88.4 ms | 69.8 ms | ~21% faster | ## What changes are included in this PR? A single-line change in `datafusion/spark/src/function/array/slice.rs` replacing `values.value(row).len() as i64` with `values.value_length(row) as i64`. ## Are these changes tested? Covered by existing unit tests in `datafusion/spark/src/function/array/slice.rs` and slt coverage for the Spark `slice` function; behavior is unchanged. The performance impact was measured with the existing `slice` criterion benchmark. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
