slachiewicz opened a new issue, #2485: URL: https://github.com/apache/iceberg-go/issues/2485
### Apache Iceberg version main (development) ### Please describe the bug 🐞 A table whose data files leave out an identity-partition source column can't be scanned. With `region` required and the table partitioned by `identity(region)`, a Parquet file that stores only `id` and carries `region` in its manifest partition value fails in `Scan().ToArrowTable`: ``` invalid schema: required field is missing and has no default value: 2: region: required string ``` Tables converted from Delta, or migrated from Hive, are written this way: the partition value lives in the path or the log, not in the file. The spec's [column projection](https://iceberg.apache.org/spec/#column-projection) resolves such a field from the partition value first: "Return the value from partition metadata if an Identity Transform exists for the field". iceberg-go fills a missing field only from `initial-default`, `write-default` or null (`arrowProjectionVisitor.Struct` in `table/arrow_utils.go`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
