mdwint commented on code in PR #2429:
URL: https://github.com/apache/iceberg-python/pull/2429#discussion_r2375133403
##########
pyiceberg/table/upsert_util.py:
##########
@@ -14,38 +14,61 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-import functools
-import operator
+from math import isnan
+from typing import Any
import pyarrow as pa
from pyarrow import Table as pyarrow_table
from pyarrow import compute as pc
from pyiceberg.expressions import (
AlwaysFalse,
+ And,
BooleanExpression,
EqualTo,
In,
+ IsNaN,
+ IsNull,
Or,
)
def create_match_filter(df: pyarrow_table, join_cols: list[str]) ->
BooleanExpression:
unique_keys = df.select(join_cols).group_by(join_cols).aggregate([])
+ filters: list[BooleanExpression] = []
if len(join_cols) == 1:
- return In(join_cols[0], unique_keys[0].to_pylist())
+ column = join_cols[0]
Review Comment:
Yes, the `In` operator cannot handle null by design, and this goes for SQL
as well.
The following SQL is invalid:
```sql
WHERE x IN (1, 2, 3, NULL)
```
Instead it should be this:
```sql
WHERE x IN (1, 2, 3) OR x IS NULL
```
Testing for null requires `IS NULL` (or `IS NOT NULL`), and it's impossible
with `IN` or `=`.
This is the reason for changing the `create_match_filter` function: we need
to build more complex expressions if null is involved. Examples of such
expressions are shown in the test cases.
If there's a better way I'm open to changing it, but I believe the added
complexity in building filter expressions with null is justified. When null is
not involved we produce the same `In` expression as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
