dwsmith1983 opened a new issue, #5801: URL: https://github.com/apache/datafusion-comet/issues/5801
### Describe the bug Comet validates Parquet field id lookups in the physical expression adapter, which resolves each file's mapping once per file (#5654). DataFusion 55's Parquet opener skips the adapter entirely when the file's physical schema compares equal to the logical schema and there is no pushed predicate. In that case a file whose struct carries two children with the same field id is read positionally instead of failing with Spark's duplicate field id error. Spark-written files always carry key-value metadata that arrow-rs folds into the physical schema, so they never compare equal and always reach the adapter. The gap is limited to files with no key-value metadata at all, read with field id matching enabled and no data filter. ### Steps to reproduce 1. Write a Parquet file without key-value metadata whose schema is `s<x: long id=1, y: long id=1>` (parquet-mr with a hand-built MessageType). 2. Read it with the native scan, `spark.sql.parquet.fieldId.read.enabled=true`, and a requested schema identical to the file schema, with no filter. 3. Spark rejects the duplicate id during schema clipping; Comet returns the rows. ### Expected behavior The duplicate field id error, as when a cast or predicate is present. ### Additional context Found in review of #5654, where the once-per-file validation was added. The short circuit is in DataFusion's opener, so the fix is either a Comet-side check before the opener decides, or an upstream option to always run the adapter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
