rich7420 opened a new issue, #7138: URL: https://github.com/apache/datafusion-comet/issues/7138
### Describe the bug Native Iceberg scans can drop rows when filtering on a column whose data files store a narrower type, for example `INT` data in a `BIGINT` column. iceberg-rust's page index skips all pages of such a file for range and `IN` predicates, and its row filter casts the literal to the file's type, so an out-of-range value becomes NULL. The post-scan filter cannot restore the rows. Such files come from type promotion and from imports by `add_files`, `migrate` or `snapshot`, which do not check file types against the schema. ### Steps to reproduce ```sql CREATE TABLE t (id INT) USING iceberg; INSERT INTO t VALUES (1), (2), (3); ALTER TABLE t ALTER COLUMN id TYPE BIGINT; INSERT INTO t VALUES (4), (5); SELECT * FROM t WHERE id > 0; ``` Reproduced at 9e1364914. ### Expected behavior All five rows, as Spark returns. Comet returns only `4` and `5`. `id < 3` and `id IN (1, 2)` return no rows. ### Additional context Imported files show the same behavior without any schema change. A file storing `id` as `INT` in a `BIGINT` column, or `t` as `TINYINT` in an `INT` column, lost rows for `id > 0`, `id IN (1, 2)`, `t < 200` and `t IN (1, 200)`; only `=` returned the right rows. Reproduced on the #6847 branch, which pushes these predicates as main does. Upstream: apache/iceberg-rust#3122. Its fix, apache/iceberg-rust#3123, was closed as stale and does not cover `TINYINT` or `SMALLINT` files. #6847 pushes only `=` and null checks on columns whose type changed in the schema history, and on every `BIGINT` and `DOUBLE` column when a residual converts only in part. Tables with a name mapping push a residual only whole. Whole residuals on imported files, or on columns promoted before Iceberg 0.12, still need the upstream fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
