mbutrovich opened a new issue, #2795:
URL: https://github.com/apache/iceberg-rust/issues/2795
## Apache Iceberg Rust version
main (also affects 0.6.0)
## Describe the bug
When a projected field is not present in a data file,
`RecordBatchTransformer` resolves it using the spec's Column Projection
fallback rules. For a field with no `initial-default`, it always produces a
null column (rule #4). This is correct for optional fields, but wrong for
required fields.
The spec's Default values section states that null is only a valid default
for an optional field:
> If either default is not set for an optional field, then the default value
is null for compatibility with older spec versions.
So a required field that is absent from a data file with no
`initial-default` has no valid value. Producing a null column for it violates
the field's required (non-nullable) constraint. In practice the transformer
builds a non-nullable Arrow array containing nulls, which fails later with a
confusing message rather than a clear error:
```
Arrow Schema Error: Invalid argument error: Column '<name>' is declared as
non-nullable but contains null values
```
Iceberg-Java raises a clear error for this case instead. In
`core/.../data/parquet/BaseParquetReaders.java` and
`spark/.../data/SparkParquetReaders.java`, the reader's field-resolution
precedence is: file value, then `initialDefault`, then null if optional, then
throw:
```java
private ParquetValueReader<?> defaultReader(Types.NestedField field,
ParquetValueReader<?> reader, int constantDL) {
if (reader != null) {
return reader;
} else if (field.initialDefault() != null) {
return ParquetValueReaders.constant(..., constantDL);
} else if (field.isOptional()) {
return ParquetValueReaders.nulls();
}
throw new IllegalArgumentException(String.format("Missing required field:
%s", field.name()));
}
```
iceberg-rust implements the first three cases but silently falls through to
null for the fourth.
## To Reproduce
Project a schema whose required field is absent from the data file and has
no `initial-default`:
```rust
// snapshot schema: required(1, "id", Int), required(2, "missing_str",
String)
// data file schema: only field id 1 ("id")
// project field ids [1, 2]
let mut transformer = RecordBatchTransformerBuilder::new(snapshot_schema,
&[1, 2]).build();
transformer.process_record_batch(file_batch_with_only_id)?; // <- produces a
null column for "missing_str"
```
Actual: the transformer builds a null column for the required `missing_str`,
which later fails with `Column 'missing_str' is declared as non-nullable but
contains null values`.
## Expected behavior
Resolution of a field absent from the data file should mirror the spec (and
Iceberg-Java): file value, then `initial-default`, then null if the field is
optional, otherwise error. A required field absent with no `initial-default`
should fail with a clear error, `Missing required field: <name>`, rather than
producing an invalid null column.
## Willingness to contribute
I can contribute a fix for this bug independently.
---
### Proposed fix
In `crates/iceberg/src/arrow/record_batch_transformer.rs`, in the "not
present" fallback (before choosing `initial_default` or null), add the
required-field case:
```rust
if iceberg_field.initial_default.is_none() && iceberg_field.required {
return Err(Error::new(
ErrorKind::DataInvalid,
format!("Missing required field: {}", iceberg_field.name),
));
}
```
Tested with a required-field-absent case (asserts the error) and an
optional-field-absent case (asserts null), pinning the precedence.
### References
- Iceberg spec, "Default values" (null is only a valid default for optional
fields) and "Column Projection" rules:
https://iceberg.apache.org/spec/#default-values
- Iceberg-Java reference behavior:
`core/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java`
and `spark/.../data/SparkParquetReaders.java` (`Missing required field`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
