manuzhang opened a new pull request, #3072:
URL: https://github.com/apache/iceberg-rust/pull/3072

   ## Which issue does this PR close?
   
   No issue is currently linked. This is the Parquet-write follow-up to #2773, 
which added the `Unknown` logical type.
   
   ## What changes are included in this PR?
   
   - Derive the physical Parquet Arrow schema from the Iceberg table schema, 
omitting `Unknown` fields. Reject schemas and container values with no writable 
physical fields, as well as unsupported Variant fields.
   - Validate incoming Arrow batches against the full logical table schema 
before projection. Reject unexpected fields and mismatched list element IDs in 
ID mode; allow Arrow's `item` list-element name in name mode.
   - Project nested structs, lists, and maps to the writable schema while 
preserving row counts. Cache the projection for repeated batch schemas.
   - Remove the precomputed `_partition` routing column before partitioned 
batches reach the file writer.
   - Count NaN values from the projected physical batch using its Arrow schema.
   
   ## Are these changes tested?
   
   - Added regressions for top-level and nested `Unknown` fields, empty 
physical schemas and containers, unsupported Variant fields, unexpected source 
fields, invalid field-ID metadata, list element identity, name matching, and 
`_partition` removal.
   - `cargo test -p iceberg --lib parquet` (57 passed)
   - `cargo test -p iceberg --lib record_batch_partition_splitter` (2 passed)
   - Focused Parquet writer tests after the final field rename (30 passed)
   - `make check-fmt`
   - `make check-clippy`
   - `git diff --check`
   
   ## AI Disclosure
   
   This PR was prepared with assistance from Codex.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to