dwsmith1983 opened a new issue, #7129: URL: https://github.com/apache/datafusion-comet/issues/7129
### Describe the bug The native Iceberg scan applies an equality delete to the wrong column when the table has two columns whose names differ only in case, such as `a` and `A`. Deleted rows come back, or the wrong rows are dropped, and which one happens can change from query to query. `planner.rs` builds every `FileScanTask` with `.with_case_sensitive(false)` (`native/core/src/execution/planner.rs:2672` on 39380cca8). In the pinned iceberg-rust (1f3bc34), the only reader of that flag outside tests binds the equality-delete predicate: `combined_predicate.bind(task.schema_ref(), task.case_sensitive())` in `crates/iceberg/src/arrow/delete_filter.rs:231`. The predicate refers to columns by name (`caching_delete_file_loader.rs`), and a case-insensitive bind resolves them through `Schema::field_by_name_case_insensitive`. That lookup is a map built from a `HashMap`, so `a` and `A` collapse to one field id, picked by iteration order. Iceberg's Java reader applies equality deletes by field id, so Spark without Comet returns the right rows. Found while reproducing #5540. ### Steps to reproduce With `spark.sql.caseSensitive=true` and the native Iceberg scan enabled: 1. Create a format-version 2 Iceberg table with columns `a INT, A INT`, and insert `(1, 20), (3, 40), (5, 60)`. 2. Commit two equality delete files through the Iceberg API: one keyed on `a` with value `1`, and one keyed on `A` with value `40`. (Spark SQL `DELETE` writes position deletes, so this needs the API, as `CometIcebergNativeSuite` does with `commitEqualityDelete`.) 3. Query the table. Spark returns only `(5, 60)`. Comet's native scan: | Query | Spark | Comet | |---|---|---| | `SELECT A ... ORDER BY A` | `[60]` | `[40], [60]` | | `SELECT a ... ORDER BY a` | `[5]` | `[1], [5]` | | `SELECT count(*)` | `[1]` | `[2]` | In the same run, the `A = 40` delete was bound to `a` in one query, and the `a = 1` delete was bound to `A` in another. ### Expected behavior The same rows as Spark: every equality delete applies to the column its field id names. ### Additional context The bound names come from the task schema: iceberg-rust evolves each delete batch to the task schema by its equality field ids before taking column names. So binding case-sensitively against the same schema matches by field id. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
