peterxcli opened a new pull request, #5794: URL: https://github.com/apache/datafusion-comet/pull/5794
## Which issue does this PR close? Part of #5550, under #5546. ## Rationale for this change Spark and Arrow can interpret the same Parquet storage differently. When reconstructing a projected Variant, these differences can change a value, reject a file Spark accepts, or select the wrong column. ## What changes are included in this PR? Use physical Parquet types for scans that project a marked Variant column, preserving ENUM as a string. Rebuild the returned footer metadata while retaining row groups, page indexes, and the original cached metadata. Normalize encoded children, unsigned integers, supported wide decimals, millisecond timestamps, fixed binary, and fixed lists before Variant reconstruction. Preserve UUID rejection, report conversion overflow, enforce field ID precedence, and reject encrypted Variant scans at the native boundary. This is a draft while empty-key metadata and Spark byte-encoding compatibility remain pending. JVM scan admission is tracked in #5551. ## How are these changes tested? Generated Parquet inputs cover physical type interpretation, ENUM/string/binary distinctions, wide decimals, schema hints, and unread Variant pruning. Native tests also cover storage normalization, overflow, field ID shadowing, encrypted scan rejection, and footer metadata preservation. Validation from `native/`: - `cargo test -p datafusion-comet --lib parquet::`: 129 passed, 1 ignored benchmark. - `cargo clippy -p datafusion-comet --lib --tests -- -D warnings`: passed. - `cargo fmt --all -- --check`: passed. `git diff --check` passed. Spark integration validation is pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
