autoCAT6 opened a new pull request, #26513: URL: https://github.com/apache/datafusion/pull/26513
**Draft: blocked on https://github.com/apache/arrow-rs/pull/11733 and a Parquet release/dependency upgrade that provides `FileEncryptionProperties::column_key_metadata`. The current stock Parquet 60.0.0 dependency cannot compile this call.** ## Which issue does this PR close? No existing issue. The reproduction, root cause and regression coverage are described below. ## Rationale for this change Converting Parquet encryption properties to DataFusion configuration can move a key identifier to a different column when only some column keys have metadata. The key bytes retain their original column associations, so direct-key file reads may still succeed while metadata-driven key retrieval requests an incorrect identifier. A mixed configuration can contain `first` with ID `first-id` and `second` with no ID. Depending on map iteration order, the converted configuration can instead put `first-id` on `second` and leave `first` without an ID. The existing `column_keys()` metadata vector skips columns without metadata, whereas its names and keys vectors include every column. Using the same position to index these vectors therefore loses the name-to-metadata association. ## What changes are included in this PR? - Retrieve each column's metadata by name using the proposed Parquet accessor. - Preserve absent metadata and explicitly empty metadata as distinct values. - Add public configuration roundtrip regressions and a real encrypted-file regression that inspects column crypto metadata and verifies exact values and nulls on readback. The draft isolates the consumer change so it can be evaluated alongside the upstream API. It needs a dependency update once the API is released; the submitted diff contains no Cargo dependency override. ## What is the testing strategy for this PR? The regression cases cover mixed, all-present and all-absent column metadata, including explicitly empty metadata and footer metadata preservation. Independent property maps exercise unspecified map iteration order. Complete property equality checks the reverse conversion; actual file assertions verify the serialized column IDs rather than relying only on successful explicit-key reads. Local validation uses Parquet 60.0.0 with only the proposed accessor added through a temporary Cargo patch. Results apply to that validation setup, not the current stock dependency: With an isolated Parquet 60 source directory containing the accessor from the upstream PR, the focused regression command is: ```sh cargo test -p datafusion-common --test encryption_metadata_roundtrip \ --features parquet_encryption \ --config 'patch.crates-io.parquet.path="/path/to/parquet-60-with-accessor"' ``` - Four new regression tests pass; the original converter fails the mixed-metadata and file-metadata tests with the same accessor-only dependency. - `datafusion-common`: 664 unit tests, four new integration tests and 81 doctests pass; five existing doctests are ignored. - Formatting, modified-file spelling, strict all-target/all-feature workspace Clippy, and the required extended workspace suite pass with the temporary override; the extended suite includes all 535 SQL test files. ## Are there any user-facing changes? Encryption property conversion preserves each column's optional key metadata. No configuration fields or existing public signatures change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
