mbutrovich opened a new issue, #3362: URL: https://github.com/apache/iceberg-rust/issues/3362
### Apache Iceberg Rust version `main` at ecba0b537a1f78ec8125c1a26f6dfab3f95b509e ### Describe the bug A table can have a data column named `_file`, `_pos`, `_deleted`, `_spec_id`, `_partition`, `_row_id` or `_last_updated_sequence_number`. These are the names of the [metadata columns](https://github.com/apache/iceberg/blob/4479136cbcd3d2f1066a01e7035aca700b4645d1/format/spec.md?plain=1#L439-L443) that a scan can project, which [`is_metadata_column_name`](https://github.com/apache/iceberg-rust/blob/ecba0b537a1f78ec8125c1a26f6dfab3f95b509e/crates/iceberg/src/metadata_columns.rs#L510-L521) accepts. When a scan selects such a column with `TableScanBuilder::select`, it resolves the name to the metadata column instead of the data column. The scan returns the metadata column's values, such as the data file's path for `_file`, with no error. [`collect_scan_field_ids`](https://github.com/apache/iceberg-rust/blob/ecba0b537a1f78ec8125c1a26f6dfab3f95b509e/crates/iceberg/src/scan/mod.rs#L66-L84) checks `is_metadata_column_name` before it looks the name up in the table schema, so the data column is never found. `select_all` doesn't go through this check, so it reads the data column. In [datafusion-iceberg](https://github.com/apache/datafusion-iceberg), which selects columns by name, `SELECT * FROM t` on a table with a `_file string` column returns file paths in that column. #2837 reported this for the bare names `pos` and `file_path`, and #2865 fixed it for those two. The reporter of #2837 hit it in a query engine built on DataFusion. #2865 made `is_metadata_column_name` match Java's `isMetadataColumn(String)`, so the remaining names are the ones Java also treats as metadata columns. Its description says: > The Iceberg spec reserves only `_`-prefixed names as metadata columns, so they can't collide with user columns. As I read the spec's [Reserved Field IDs](https://github.com/apache/iceberg/blob/4479136cbcd3d2f1066a01e7035aca700b4645d1/format/spec.md?plain=1#L439-L443) section, it reserves the field ids above 2147483447 for metadata columns, not their names. Iceberg Java allows a data column with one of these names, and it handles the conflict in two ways depending on the API: - In core, [`BaseScan.lazyColumnProjection`](https://github.com/apache/iceberg/blob/4479136cbcd3d2f1066a01e7035aca700b4645d1/core/src/main/java/org/apache/iceberg/BaseScan.java#L267-L295) resolves the names passed to `select` with `schema.select`, against the table schema only, so a data column named `_file` is read. A caller that wants a metadata column adds it to a projected `Schema` passed to `Scan.project`, where it is identified by field id, so the names never collide. - Spark selects metadata columns by name, as iceberg-rust's `select` does. [`BaseSparkScanBuilder.pruneColumns`](https://github.com/apache/iceberg/blob/4479136cbcd3d2f1066a01e7035aca700b4645d1/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/BaseSparkScanBuilder.java#L132-L145) treats every requested name that `isMetadataColumn` accepts as a metadata column. [`SparkScan`](https://github.com/apache/iceberg/blob/4479136cbcd3d2f1066a01e7035aca700b4645d1/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java#L136) then calls [`SparkSchemaUtil.validateMetadataColumnReferences`](https://github.com/apache/iceberg/blob/4479136cbcd3d2f1066a01e7035aca700b4645d1/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkSchemaUtil.java#L343-L357), which throws a `ValidationException` when the read references a metadata column name that the table schema also has. This came from apache/iceberg#3456. [`TestSparkMetadataColumns.testConflictingColumns`](http s://github.com/apache/iceberg/blob/4479136cbcd3d2f1066a01e7035aca700b4645d1/spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkMetadataColumns.java#L277-L310) adds `_spec_id` and `_file` data columns and checks that `SELECT id, category` works, that `SELECT *` fails with `Table column names conflict with names reserved for Iceberg metadata columns: [_spec_id, _file].`, and that both columns can be read after they are renamed. ### To Reproduce #2865 added a helper, [`table_with_data_column`](https://github.com/apache/iceberg-rust/blob/ecba0b537a1f78ec8125c1a26f6dfab3f95b509e/crates/iceberg/src/scan/mod.rs#L2292-L2350), that builds a table whose schema is `1: id int` and `2: <name> int`, with no files on disk. This test, next to [`test_scan_projects_data_column_named_like_delete_file_column`](https://github.com/apache/iceberg-rust/blob/ecba0b537a1f78ec8125c1a26f6dfab3f95b509e/crates/iceberg/src/scan/mod.rs#L2352-L2386), shows the problem: ```rust #[test] fn test_scan_projects_data_column_named_like_metadata_column() { use crate::metadata_columns::*; for column_name in [ RESERVED_COL_NAME_FILE, RESERVED_COL_NAME_POS, RESERVED_COL_NAME_DELETED, RESERVED_COL_NAME_SPEC_ID, RESERVED_COL_NAME_PARTITION, RESERVED_COL_NAME_ROW_ID, RESERVED_COL_NAME_LAST_UPDATED_SEQUENCE_NUMBER, ] { let table = table_with_data_column(column_name); let table_scan = table.scan().select([column_name]).build().unwrap(); assert_eq!( table_scan.plan_context.as_ref().unwrap().field_ids.as_ref(), &[2] ); } } ``` On `main` it fails on the first name, because `_file` resolves to the metadata column's reserved field id: ``` assertion `left == right` failed left: [2147483646] right: [2] ``` The other six names resolve to their reserved field ids in the same way. ### Expected behavior The scan should not return a metadata column's values for a data column. Should `TableScanBuilder::build` return an error when a selected name is both a metadata column name and a field of the table schema, as Spark does? Iceberg Java settles this case because the spec says nothing about names, and its name-based API raises an error. Letting the data column win, as Java core does, would leave the metadata column unreachable on that table, because `select` has no other way to ask for it. With an error, engines that know the field ids still couldn't read a table that has such a column. Would an additive way to select columns by field id, similar to Java's `Scan.project`, be worth a separate feature request? datafusion-iceberg already has the field id of every column in its Arrow schema, so it could read the data column with no name lookup. ### Willingness to contribute I would be willing to contribute a fix for this bug with guidance from the Iceberg community -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
