fresh-borzoni opened a new pull request, #7419: URL: https://github.com/apache/datafusion-comet/pull/7419
## Which issue does this PR close? Closes #7417. ## Rationale for this change Pick up the fixes in arrow-rs 59.3.2. It is a patch release over 59.3.1 with 15 backported fixes. The changes most relevant to Comet are: - `DeltaBitPackDecoder::skip` reads `DELTA_BINARY_PACKED` miniblocks wider than 64 values instead of failing with `cannot skip miniblock of size N` (apache/arrow-rs#11754, the 59.x backport of apache/arrow-rs#11021). The native Iceberg scan hits that error when position deletes make it skip a long run of rows in files written by Snowflake or Databricks Photon (#7417). - The IPC writer writes empty validity buffers for arrays without nulls (apache/arrow-rs#11820). - `page_encoding_stats` lists the dictionary page first (apache/arrow-rs#11699). - A use-after-free in `Buffer::shrink_to_fit` when `MemoryReservation::resize` panics is fixed (apache/arrow-rs#11787). The rest are `ArrayData` and `MutableArrayData` fixes, mostly for union and run-end-encoded arrays. Full changelog: https://github.com/apache/arrow-rs/blob/59.3.2/CHANGELOG.md ## What changes are included in this PR? - Refresh the native and Delta contrib lockfiles with `cargo update -p arrow --precise 59.3.2` and `cargo update -p parquet --precise 59.3.2`. This moves the `arrow-*`, `parquet` and `parquet-variant*` crates from 59.3.1 to 59.3.2. - No `Cargo.toml` change: the `59.2.0` requirement already admits 59.3.2. DataFusion, iceberg-rust and object_store remain unchanged. `native/native-executors` is left out; #7415 bumps its `arrow`. ## How are these changes tested? Locally verified on macOS/aarch64: - `cargo check --workspace --all-targets --locked` - `cargo clippy --workspace --all-targets --locked -- -D warnings` - `cargo fmt --all -- --check` - `cargo check -p datafusion-comet --features contrib-delta --locked` - `cargo check --locked --all-targets` in `contrib/delta/native` The reproduction in #7417 fails on Comet 1.1.0. I built 1.2.0-rc1 with this lockfile change as a Spark 3.5 jar for linux/aarch64 and linux/x86_64. That build returns Spark's results for the query, and `compute_table_stats` writes the same NDVs as Spark. arrow-rs covers the decoder change with `test_delta_bit_packed_skip_wide_miniblocks` and `non_standard_delta_blocks`. Could a committer add `run-iceberg-tests` and `run-delta-build-gate`? The decoder sits under the native Iceberg scan, and the Delta contrib lockfile changes too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
