andygrove opened a new pull request, #6579: URL: https://github.com/apache/datafusion-comet/pull/6579
## Which issue does this PR close? Backport of #6339 to `branch-1.0`. The original PR closes #6288. ## Rationale for this change Arrow Java's C Data import ignores `ArrowArray.offset` at every level. arrow-rs folds a slice into the buffers for every Spark type except boolean, which keeps its bit offset, so the JVM can read sliced boolean values and nulls from bit 0. The existing fix only checks the top-level output column and therefore misses booleans nested in structs, lists, or maps, as well as inputs to the JVM UDF bridge. ## What changes are included in this PR? - Add `zero_offsets` to normalize offsets recursively before arrays cross the native-to-JVM boundary. - Apply normalization in `move_to_spark` and the JVM UDF bridge, replacing the top-level `take` workaround. - Add native unit coverage and six Scala regressions for sliced booleans in aggregate, offset, UDF, dispatched-expression, and `collect_list` paths. - Document the export requirement in the FFI contributor guide. - Adapt the implementation to branch-1.0's Arrow API and existing two-argument `move_to_spark` API. ## How are these changes tested? - `cargo test --manifest-path native/Cargo.toml -p datafusion-comet-common ffi_offsets --lib` — 7 passed. - `cargo test --manifest-path native/Cargo.toml -p datafusion-comet test_move_to_spark_zeroes_nested_boolean_offsets --lib` — 1 passed. - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometCodegenSuite,org.apache.comet.exec.CometExecSuite" -Dscalastyle.skip=true` — 236 passed across 2 suites. - `cargo fmt --manifest-path native/Cargo.toml --all -- --check`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
