slachiewicz opened a new issue, #4267: URL: https://github.com/apache/iceberg-python/issues/4267
### Apache Iceberg version 0.12.0 (latest release) ### Please describe the bug 🐞 The spec's [Column Projection](https://iceberg.apache.org/spec/#column-projection) rules say a field missing from a data file takes its value from the partition struct when an identity transform exists for it. #1401 implemented this, but only for optional fields. For a `required` field, `ArrowProjectionVisitor.struct` raises before it checks `_projected_missing_fields`: ``` pyiceberg.exceptions.ResolveError: Field is required, and could not be found in the file: 2: region: required string ``` This affects tables written by engines that keep identity partition columns out of the data files, for example Hive-style layouts and tables converted from Delta, where partition columns are never in the Parquet files. Reproduction (SQL catalog, local warehouse): ```python import os, tempfile import pyarrow as pa, pyarrow.parquet as pq from pyiceberg.catalog.sql import SqlCatalog from pyiceberg.schema import Schema from pyiceberg.types import NestedField, LongType, StringType from pyiceberg.partitioning import PartitionSpec, PartitionField from pyiceberg.transforms import IdentityTransform from pyiceberg.manifest import DataFile, DataFileContent, FileFormat from pyiceberg.typedef import Record wh = tempfile.mkdtemp() cat = SqlCatalog("c", uri=f"sqlite:///{wh}/c.db", warehouse=f"file://{wh}") cat.create_namespace("db") schema = Schema(NestedField(1, "id", LongType(), required=True), NestedField(2, "region", StringType(), required=True)) spec = PartitionSpec(PartitionField(source_id=2, field_id=1000, transform=IdentityTransform(), name="region")) tbl = cat.create_table("db.t", schema=schema, partition_spec=spec) path = f"{wh}/region=north/f.parquet" os.makedirs(os.path.dirname(path)) pq.write_table(pa.table({"id": pa.array([1, 2], pa.int64())}, schema=pa.schema([pa.field("id", pa.int64(), nullable=False, metadata={"PARQUET:field_id": "1"})])), path) df = DataFile.from_args(content=DataFileContent.DATA, file_path=f"file://{path}", file_format=FileFormat.PARQUET, partition=Record("north"), record_count=2, file_size_in_bytes=os.path.getsize(path)) df.spec_id = 0 with tbl.transaction() as tx: with tx.update_snapshot().fast_append() as app: app.append_data_file(df) print(tbl.scan().to_arrow()) # ResolveError ``` With `region` declared optional, the same script returns `region: [["north","north"]]`. ### Willingness to contribute - [x] I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
