potatochipcoconut commented on issue #1776:
URL:
https://github.com/apache/iceberg-python/issues/1776#issuecomment-2895450565
FYI tried this out in distributed write scenario where data files are
serialized, then deserialized as described in link above, this is result when
trying to do `txn.upsert()`
```
[ERROR] AttributeError: 'DataFile' object has no attribute 'select'
Traceback (most recent call last):
File "/var/task/write/lambda_function.py", line 26, in handler
write_to_iceberg(data_files)
File "/var/task/ocr_experiments/utils.py", line 317, in write_to_iceberg
write_data(table, data)
File "/var/task/ocr_experiments/utils.py", line 329, in write_data
txn.upsert(data_file, join_cols=['id'])
File "/var/lang/lib/python3.12/site-packages/pyiceberg/table/__init__.py",
line 763, in upsert
if upsert_util.has_duplicate_rows(df, join_cols):
File
"/var/lang/lib/python3.12/site-packages/pyiceberg/table/upsert_util.py", line
53, in has_duplicate_rows
return len(df.select(join_cols).group_by(join_cols).aggregate([([],
"count_all")]).filter(pc.field("count_all") > 1)) > 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
