manuzhang opened a new pull request, #3072: URL: https://github.com/apache/iceberg-rust/pull/3072
## Which issue does this PR close? No issue is currently linked. This is the Parquet-write follow-up to #2773, which added the `Unknown` logical type. ## What changes are included in this PR? - Derive the physical Parquet Arrow schema from the Iceberg table schema, omitting `Unknown` fields. Reject schemas and container values with no writable physical fields, as well as unsupported Variant fields. - Validate incoming Arrow batches against the full logical table schema before projection. Reject unexpected fields and mismatched list element IDs in ID mode; allow Arrow's `item` list-element name in name mode. - Project nested structs, lists, and maps to the writable schema while preserving row counts. Cache the projection for repeated batch schemas. - Remove the precomputed `_partition` routing column before partitioned batches reach the file writer. - Count NaN values from the projected physical batch using its Arrow schema. ## Are these changes tested? - Added regressions for top-level and nested `Unknown` fields, empty physical schemas and containers, unsupported Variant fields, unexpected source fields, invalid field-ID metadata, list element identity, name matching, and `_partition` removal. - `cargo test -p iceberg --lib parquet` (57 passed) - `cargo test -p iceberg --lib record_batch_partition_splitter` (2 passed) - Focused Parquet writer tests after the final field rename (30 passed) - `make check-fmt` - `make check-clippy` - `git diff --check` ## AI Disclosure This PR was prepared with assistance from Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
