peterxcli opened a new issue, #5550: URL: https://github.com/apache/datafusion-comet/issues/5550
### Problem Arrow-rs and Spark can derive different Arrow types and name matches from the same Parquet file. For projected Variant storage this can change the reconstructed value, reject a file Spark reads, or select the wrong physical column. Parent: #5546 Depends on: #5549 Cleanup trackers: #5474, #5477, #5495 ### Solution At the projected, explicitly marked Variant boundary only: - Prevent advisory `ARROW:schema` types from overriding Spark's physical Parquet interpretation. - Preserve Parquet ENUM as a Variant string while leaving raw BINARY as binary. - Match Spark's physical decimal, date/timestamp, fixed binary/list, and encoded child behavior needed by whole-value reconstruction. - Preserve row-group metadata and page/offset indexes when rebuilding reader metadata. - Keep encrypted affected scans on Spark when reader metadata cannot be reconstructed safely. - Match Spark field-ID precedence and fail closed for unresolved Unicode/name ambiguity without duplicating the generic work in #5495. The JVM Variant scan gate remains closed in this task. ### Tests Use generated Parquet inputs rather than committed fixtures for: - encoded storage children and supported unsigned widths; - timestamp, fixed binary, and fixed-size-list storage; - positive and negative wide DECIMAL values; - ENUM versus UTF8 versus raw BINARY; - advisory Arrow-schema hints; - field-ID shadowing and Unicode/name ambiguity; - encrypted fallback; and - pruning that does not activate compatibility work for an unread Variant column. Each compatibility branch must link its upstream owner and the corresponding cleanup tracker. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
