GaspardMerten opened a new issue, #3860:
URL: https://github.com/apache/iceberg-python/issues/3860
### Feature Request / Improvement
`upsert` compares the matched rows one cell at a time, so it gets slower
with every extra column, not just with every extra row.
On a table with 200 columns, comparing 20k matched rows takes around 20
seconds on my machine, before anything is written.
To reproduce:
```python
import time
import pyarrow as pa
from pyiceberg.table.upsert_util import get_rows_to_update
rows, cols = 20_000, 200
table = pa.table({"pk": pa.array(range(rows)), **{f"c{i}":
pa.array([float(i)] * rows) for i in range(cols)}})
start = time.monotonic()
get_rows_to_update(table, table, ["pk"]) # nothing has changed
print(time.monotonic() - start)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
