brgr-s opened a new issue, #3369: URL: https://github.com/apache/iceberg-rust/issues/3369
### Apache Iceberg Rust version 0.10.1 ### Describe the bug This if a follow up for #2961, but it also applies if it is not merged The spec says ([format/spec.md](https://github.com/apache/iceberg/blob/212aa894e919ff9fdf89160da4aadcbe6858ac01/format/spec.md?plain=1#L1466)): > If a delete column in an equality delete file is later dropped from the table, it must > still be used when applying the equality deletes. If a column was added to a table and > later used as a delete column in an equality delete file, the column value is read for > older data files using normal projection rules (defaults to `null`). **Case A: delete column later dropped from the table.** Equality-delete files are evolved against `task.schema`, the current table schema. A dropped column is not in that schema. The scan fails in `RecordBatchTransformer::generate_batch_transform` with `"field not found"` (`ErrorKind::Unexpected`), instead of using the dropped column for matching. Java's `DeleteFilter` seems to fail the same way ([DeleteFilter.java:76](https://github.com/apache/iceberg/blob/main/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L76), [DeleteFilter.java:318](https://github.com/apache/iceberg/blob/main/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L318)). **Case B: delete column added after the data file was written.** "Normal projection rules" means the reader should use the field's `initial_default`. The equality-delete probe runs inside the parquet `RowFilter`, on raw decoded batches, before `RecordBatchTransformer` resolves defaults. So a key column that is missing from the data file is always read as `null`, never as `initial_default`. Example: a required column `added` with `initial_default = 7` is added to the table. An equality-delete file keyed on `added` deletes `added = 7`. Rows in data files written before `added` existed should be deleted. They are not, because they probe as `null`. This is a silent wrong result, not an error. It is pinned by `test_eq_delete_on_column_absent_from_data_file` (`crates/iceberg/src/arrow/reader/row_filter.rs`). Before #2961, the code was also wrong, in a different way: a missing *required* key column deleted every row of the data file, and a missing *optional* key column deleted nothing. ### To Reproduce **Context:** Found during review of #2961, which does not change these paths. See https://github.com/apache/iceberg-rust/pull/2961#discussion_r3722140834 and https://github.com/apache/iceberg-rust/pull/2961#issuecomment-5205187279. ### Expected behavior - Case A: a dropped column that an equality-delete file references is still used for matching. - Case B: a key column missing from an older data file is read with normal projection rules (its `initial_default`, else `null`) before the delete-set lookup. ### Willingness to contribute I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
