mbutrovich opened a new pull request, #2797: URL: https://github.com/apache/iceberg-rust/pull/2797
## Which issue does this PR close? - Closes #2795. ## What changes are included in this PR? When a projected field is not present in a data file, `RecordBatchTransformer` resolves it using the spec's Column Projection fallback rules. For a field with no `initial-default` it produced a null column (rule #4) unconditionally. That is correct only for optional fields. Per the spec's Default values section, null is a valid default only for an optional field: > If either default is not set for an optional field, then the default value is null for compatibility with older spec versions. So a required field that is absent from a data file with no `initial-default` has no valid value, and producing a null column for it violates the field's required (non-nullable) constraint. Previously this surfaced later as a confusing Arrow error (`Column '<name>' is declared as non-nullable but contains null values`) instead of a clear one. This PR adds the required-field case to the "not present" fallback in `record_batch_transformer.rs`: if a field is absent from the data file, has no `initial-default`, and is required, return `Missing required field: <name>`. The resolution order now mirrors Iceberg-Java's Parquet readers (`BaseParquetReaders` / `SparkParquetReaders.defaultReader`): file value, then `initial-default`, then null if optional, otherwise error. ## Are these changes tested? Yes, unit tests in `record_batch_transformer.rs`. Together with the existing `test_all_four_spec_rules`, all four combinations of a field absent from the data file are covered: - absent + has `initial-default` returns the default value (rule #3) - existing `test_all_four_spec_rules`. - absent + optional + no default returns null (rule #4) - existing `test_all_four_spec_rules`, plus a new focused `schema_evolution_optional_field_absent_without_default_is_null`. - absent + required + no default now errors with `Missing required field: <name>` - new `schema_evolution_required_field_absent_without_default_errors`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
