tomron opened a new pull request, #4210:
URL: https://github.com/apache/iceberg-python/pull/4210
Closes #4203
## Rationale for this change
`_task_to_record_batches` re-bound the row filter to each data file's own
schema, which converted the literal to the file's older type. After
`update_column("f", DoubleType())` on a `float` column, `0.1` became the 32-bit
`0.1`, so `f > 0.1` and `f != 0.1` dropped rows written before the promotion
even though the scan returns them as `0.10000000149011612`. A widened decimal
(`decimal(5,2)` to `decimal(12,2)`) raised `ArrowInvalid` for the same reason.
The filter now stays bound to the table schema and Arrow compares the mixed
types. This matches `delete()`, `upsert` and the stats-based file pruning,
which already compare in the table's type. Previously `scan` and `delete`
disagreed on which rows matched.
## What changed
- Added `resolve_missing_columns`, which replaces predicates on columns
missing from the file with constants (projected partition value, then
`initial-default`, as before) and leaves the other predicates bound to the
table schema. It shares its logic with `_ColumnNameTranslator`, and
`translate_column_names` is unchanged.
- `_task_to_record_batches` uses it and passes `file_schema` to
`expression_to_pyarrow`, which resolves column names by field ID.
## Not in this PR
`IN` / `NOT IN` on a promoted float still compares in the file's type,
because Arrow casts the `isin` value set to the column type. That is unchanged
from before and can be a follow-up.
## Are these changes tested?
Yes. New tests cover float to double, decimal widening and int to long. They
also cover nested, renamed and later-added columns, files from before and after
the promotion in one scan, nulls, positional deletes, partition-projected and
`initial-default` columns, and `delete()` matching `scan`. They fail without
the fix.
## Are there any user-facing changes?
Row filters on columns whose type was promoted now return the same rows as
the promoted values, instead of dropping rows from older files or raising.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
