eneskeles opened a new issue, #2132:
URL: https://github.com/apache/iceberg-go/issues/2132
### Apache Iceberg version
None
### Please describe the bug 🐞
Follow-up from #2111.
Whether `NotIn` matches a NULL row depends on how many distinct values are
in the set.
Scan on one data file with `age = [25, 30, NULL, 40]`:
| filter | rows returned |
|---|---|
| `NotIn(age, 30, 40)` | `25`, `NULL` |
| `NotIn(age, 30, 30)` | `25`, `40` |
| `NotEqualTo(age, 30)` | `25`, `40` |
### Why
Binding turns a set with one distinct value into `NotEqual`, which is NULL
for a NULL term, so the row is not returned. A set with more values stays
`NotIn` and runs as `not(is_in(x, {...}))`, which is true for NULL.
### Effect on deletes
After #2111, a copy-on-write `Delete(NotIn(age, 30, 40))` keeps the NULL
row, because the rewrite keeps every row where the filter is not true and
treats `NotIn` like `NotEqual`. A scan with the same filter returns that row,
and merge-on-read deletes it. So the two delete modes disagree for multi-valued
`NotIn`.
### Expected
`NotIn` should treat NULL the same way for any number of values, and scan,
copy-on-write and merge-on-read should agree. I think the SQL behaviour (NULL
never matches) is the natural one since `NotEqual` already works that way, but
I'm happy to go with whatever matches Java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
