cayetanobv commented on PR #3624:
URL: https://github.com/apache/iceberg-python/pull/3624#issuecomment-6062769218
Tested this branch (`c152089da`) against the Apache REST catalog fixture
1.10.1. The manifest work does what it says: `ManifestWriterV3` and
`ManifestListWriterV3` are there and `write_manifest` dispatches on version 3.
`add_files` to a v3 table still fails, but one step later than before.
```
CommitStateUnknownException: RESTError 500
```
The server names the cause:
```
java.lang.IllegalArgumentException: Invalid added-rows (required when
first-row-id is set): null
```
`_SnapshotProducer._commit` sets `first_row_id` and not `added_rows`
(`table/update/snapshot.py`, lines 337-361 on this branch):
```python
first_row_id: int | None = None
if self._transaction.table_metadata.format_version >= 3:
first_row_id = self._transaction.table_metadata.next_row_id
...
snapshot = Snapshot(
snapshot_id=self._snapshot_id,
...
first_row_id=first_row_id,
)
```
`_calculate_added_rows` is defined at line 212 in the same class and has no
caller, so the `AddSnapshotUpdate` carries `first-row-id` without `added-rows`.
Reproduction, against the REST fixture with an S3 warehouse:
```python
from pyiceberg.catalog import load_catalog
import pyarrow as pa, pyarrow.parquet as pq, tempfile, pathlib
cat = load_catalog("t", **{"type": "rest", "uri": "http://localhost:8181";,
...})
d = pathlib.Path(tempfile.mkdtemp()); src = d / "data.parquet"
pq.write_table(pa.table({"id": pa.array([1, 2, 3], type=pa.int64())}), src)
t = cat.create_table("ns.reg", schema=..., properties={"format-version":
"3"})
t.add_files([f"file://{src}"]) # RESTError 500
```
I am glad to open a follow-up with the assignment and a test, or it may
belong in this pull request.
*Context: we are evaluating version 3 geometry tables for
[Portolan](https://www.portolan-sdi.org/), which is why we exercise `add_files`
on version 3 rather than `append`.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
