Rahul-cs50 opened a new pull request, #4219: URL: https://github.com/apache/iceberg-python/pull/4219
Closes #4184 # Rationale for this change When `delete()` (and `overwrite()` with an `overwrite_filter`) has to rewrite a data file, it keeps the rows that `_expression_to_complementary_pyarrow` selects. That filter was `NOT f`, OR'd with `c IS NULL` for every column `c` that `f` reads without a null check. Under pyarrow's three-valued logic that goes wrong both ways when a column is null: - `NOT f` is null where `f` is null, so the row is dropped even though `f` did not match it: x=2 in the issue (`p == 2`, `v` null) is deleted by `(p == 1 AND v IS NULL) OR (p == 2 AND v == 'c')`. - `c IS NULL` keeps a row whenever `c` is null, even where `f` is still true: `p == 1 OR v == 'c'` keeps x=1 (`p == 1`, `v` null). A row should be deleted exactly where `f` is true, which are the rows `scan(row_filter=f)` returns. The complement is now `NOT coalesce(f, false)`. The NaN handling (`OR is_nan(c)` for columns the filter does not NaN-check) is left as it was. ## Are these changes tested? Yes. `test_delete_and_overwrite_filter_with_null` in `tests/catalog/test_catalog_behaviors.py` runs the issue's two filters plus `v != 'c'` and `NOT (v == 'c')` through `delete()` and `overwrite()` on every local catalog, and checks that the rows left are the ones `scan(row_filter=...)` does not return. It fails on `main` and passes with this change. `test_expression_to_complementary_pyarrow` is updated for the new expression. Unit tests in `tests/io`, `tests/table`, `tests/catalog` and `tests/expressions`: 2658 passed (the only failures are the four Kerberos tests in `test_hive.py`, which need the `hive-kerberos` extra I did not install). `ruff check` and `ruff format --check` are clean. ## Are there any user-facing changes? `delete()` and `overwrite(..., overwrite_filter=...)` no longer delete rows the filter does not match, or keep rows it does, when a column the filter reads is null. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
