andygrove opened a new issue, #6993: URL: https://github.com/apache/datafusion-comet/issues/6993
### Describe the bug A `VERSION AS OF` read of a column whose type was promoted after the snapshot (int to bigint, float to double, a wider decimal, including a nested field) fails in the native Iceberg scan. Spark returns the snapshot's values. ```sql CREATE TABLE t (id INT, x INT) USING iceberg; INSERT INTO t VALUES (1, 10); -- take snapshot S here ALTER TABLE t ALTER COLUMN x TYPE BIGINT; SELECT id, x FROM t VERSION AS OF S; ``` ``` org.apache.comet.CometNativeException: Batch adaptation failed: External error: Parquet column cannot be converted in file . Column: [[x]], Expected: int, Found: INT64 ``` Reproduced on `main` at 7d8d9b1544 (Spark 4.1, Iceberg 1.11). `branch-1.1` has the same task-schema code. Spark reads a time travel query with the snapshot's schema, so `x` is `int`. The native task reads with the current table schema instead: `tableSchema` for a task without deletes, and `task.schema()` for a task with deletes. iceberg-rust then returns `x` as `bigint`, and the batch adaptation refuses to narrow it back. When a projected column was dropped after the snapshot, a task without deletes reads the scan schema instead, and the query works. #6725 (open) makes the native scan read the pruned scan schema by default, which fixes this for scans that can prune. `branch-1.1` needs its own fix. ### Steps to reproduce See above. Tasks with position deletes, an identity partition source projected after its promotion, and a promoted nested field (`s.a`) fail the same way. ### Expected behavior The native scan returns the snapshot's values with the snapshot's types, as Spark does. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
