jatorre opened a new pull request, #3642:
URL: https://github.com/apache/iceberg-rust/pull/3642
## Which issue does this PR close?
- Closes #3639. Stacked on #3641 (its first commit); only the second commit
is new here.
## What changes are included in this PR?
`pyiceberg_core.transaction.add_files(identifier, metadata_location,
table_metadata, file_paths, io_properties, snapshot_properties=None) -> str`
It reads each file's footer, writes the manifest and manifest list next to
the table, and returns the commit as JSON (`{"requirements": [...], "updates":
[...]}`). The caller sends it with its own catalog, for PyIceberg:
```python
body = json.loads(transaction.add_files(list(ident),
table.metadata_location, raw_metadata_json, paths, io_props))
reqs = tuple(TypeAdapter(TableRequirement).validate_python(r) for r in
body["requirements"])
ups = tuple(TypeAdapter(TableUpdate).validate_python(u) for u in
body["updates"])
catalog.commit_table(table, reqs, ups)
```
This gives PyIceberg add-files on format-version 3 tables (with row lineage)
before it writes v3 snapshots itself. Unpartitioned tables only (refused
otherwise), like `parquet_files_to_data_files`.
## Are these changes tested?
`tests/test_transaction.py`: a v3 append of two local Parquet files returns
`assert-table-uuid`/`assert-ref-snapshot-id` and
`add-snapshot`/`set-snapshot-ref` with `first-row-id` 0, `added-rows` 5, the
snapshot properties, and a written manifest list; a partitioned table is
refused. The binding's suite passes (43, 1 xfailed).
End to end, with the snippet above: PyIceberg 0.12's REST client committed
it to a v3 table with a `geometry` column; DuckDB 2.0 and PyIceberg read 3,000
rows with consistent row ids (`next-row-id` 3000), and a later DuckDB insert
continued them (3010). PyIceberg's scan of that table also needs #3640
(manifest codec) or apache/iceberg-python#4295.
## AI Disclosure
Drafted with Claude Code (Claude Opus 5.5) at my direction; I ran the tests,
clippy, rustfmt, ruff, and the end-to-end check described above. Points I would
like reviewers to look at:
- **Size and scope.** The binding now depends on `iceberg-storage-opendal`
(and tokio's runtime): the manylinux aarch64 wheel goes from 1.5 MB (main) to
5.1 MB. An alternative is to take the footers and write the manifests through a
Python `FileIO`; it keeps the wheel small but needs a FileIO bridge. Which do
you prefer?
- **JSON as the interface** rather than pyclasses: it matches the REST
request PyIceberg already builds, but I can return typed objects instead.
- One current-thread tokio runtime per call, with the GIL released
(`py.detach`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
