dwsmith1983 opened a new pull request, #7130: URL: https://github.com/apache/datafusion-comet/pull/7130
## Which issue does this PR close? Closes #5540. Closes #7129. The Unicode part of #5540 was fixed earlier on main: names are now folded with the JVM's case data (`name_fold.rs`). What remained is that the native Iceberg scan matched names case-insensitively at all, which #5540 also covers and which this PR removes. ## Rationale for this change The native Iceberg scan matched column names case-insensitively in two places, though both sides of each match already carry the table schema's exact names: - The schema adapter (`iceberg_scan.rs`) used `SparkParquetOptions` with `case_sensitive` left at `false`. A table with columns `a` and `A`, which Iceberg allows, failed with `[_LEGACY_ERROR_TEMP_2093] Found duplicate field(s) "a": [[a, A]]` on any scan that read both. Spark reads it. - `planner.rs` built every `FileScanTask` with `.with_case_sensitive(false)`. In iceberg-rust that flag only binds equality-delete predicates, and a case-insensitive bind resolves `a` and `A` to one field id picked by `HashMap` order. Equality deletes then hit the wrong column: deleted rows came back, and which delete went wrong changed between queries (#7129). Exact matching is right on both sides, whatever `spark.sql.caseSensitive` says: - iceberg-rust relabels every batch to the task schema's names by field id, name-mapped files included. - Spark's scan attributes carry the table's names, not the case the query was written in. - iceberg-rust evolves each equality-delete batch to the task schema by its equality field ids before it takes column names, so binding them exactly binds by field id. Iceberg's Java reader applies equality deletes by field id too. ## What changes are included in this PR? - The Iceberg schema adapter matches names exactly (`iceberg_parquet_options`). - `FileScanTask`s are built with `case_sensitive` set, so equality deletes bind exactly. - A top-level column missing from the batch iceberg-rust returns is now an error instead of a column of nulls. iceberg-rust always returns every projected column, so this only fires on a name mismatch, which matching names exactly would otherwise turn into silent nulls. Nested fields are unchanged; a nested rename that differs only in case already falls back to Spark. - The user guide no longer lists the duplicate-field failure. ## How are these changes tested? New tests in `CometIcebergNativeSuite`, each compared with Spark: - With `spark.sql.caseSensitive=true`, reads of a table with `a` and `A`, filters on each, and equality deletes keyed on each. These fail on main, with the duplicate-field error or with deleted rows returned. - With `spark.sql.caseSensitive=false`, a read of that table with its equality deletes. This fails on main on Spark 3.5 and 4.0. Spark 4.1 rejects the read itself, and the test checks that. - Guards for the exact matching, which pass on main as well: mixed-case names queried in another case, an equality delete after a case-only column rename, a name-mapped file whose physical names differ in case, and a nested case-only rename, which falls back to Spark. Rust unit tests cover the adapter telling `a` from `A` and the missing-column error. Reverting the planner change fails the equality-delete test; reverting the adapter change fails the read tests and the unit test. `CometIcebergNativeSuite` passes on Spark 3.5, 4.0 and 4.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
