Revanth14 opened a new pull request, #2356:
URL: https://github.com/apache/iceberg-go/pull/2356
Closes #2132
## Problem
A multi-valued `NotIn` runs as `not(is_in(x, {...}))`. Arrow's `is_in` is
false for a NULL input, so `NotIn` matched NULL rows, while a one-value set
binds to `NotEqual`, which doesn't. Scans and merge-on-read deletes followed
the row filter, while copy-on-write rewrites already treated `NotIn` as NULL
for NULL, so the delete modes disagreed. `NOT(In(...))` had the same problem.
A column added after a file was written had a similar split: translation
folded any predicate on it to false, so `NOT(x = 5)` or `NOT(x IN (5, 6))`
matched every row of that file, while a stored NULL matched neither.
## Fix
`In` is now NULL for a NULL value, as SQL `IN` and `Equal` are: the row
filter is `or(is_in(x, s), and(is_null(x), NULL))`. The second term is false
for any other value, so the result is unchanged there. `NotIn` negates it, so
it never matches NULL either, for any number of values. Arrow's set-lookup null
behavior can't be chosen through the substrait mapping (it is fixed to
`NullMatchingMatch`), and and-ing with `is_not_null` would give false rather
than NULL, so `NOT(NotIn(...))` would still match NULL rows.
For a missing column, a predicate that is NULL for a null value now folds to
false when no NOT applies and to true under an odd number of NOTs, which gives
the same rows as a stored NULL under three-valued logic. `IsNull`, `NotNull`,
`IsNaN` and `NotNaN` keep their current folds.
This diverges from Java's row `Evaluator`, where `notIn(null)` is true.
## Testing
- `TestInAndNotInNeverMatchNull`: the issue's data through a scan and both
delete modes, unpartitioned and with an identity partition on the filtered
column, for `In`, `NotIn`, one-value sets, and `In` under NOT.
- `TestMissingColumnMatchesLikeStoredNull`: one table with rows from a file
written before the column existed and a stored NULL, which must match alike
under NOT, through a scan and both delete modes.
- `TestNanosecondTimestampLiterals` is updated for the new expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
