kotwal-itpro opened a new pull request, #4123:
URL: https://github.com/apache/iceberg-python/pull/4123
Closes #4095
# Rationale for this change
The REST spec's `POST /v1/{prefix}/transactions/commit` applies changes to
several tables in one atomic operation, and the Java client exposes it as
`RESTCatalog.commitTransaction(...)`. PyIceberg had no way to call it, so
write-audit-publish across tables or a backfill of related tables couldn't be
published atomically from Python.
This adds `Catalog.commit_transactions(*transactions)`:
```python
orders_tx = orders.transaction()
orders_tx.append(new_orders)
items_tx = items.transaction()
items_tx.append(new_items)
catalog.commit_transactions(orders_tx, items_tx) # both tables, or neither
```
### What it does
- **REST catalog:** sends each transaction's requirements and updates as one
`CommitTransactionRequest`, adding the same `AssertTableUUID` that
`commit_transaction` adds. The endpoint returns 204, so each table is refreshed
afterwards and the transactions are cleared.
- **Other catalogs:** the base `Catalog` raises `NotImplementedError`, so
they can add support later.
- **Errors** map like `commit_table`: 409 → `CommitFailedException`, 404 →
`NoSuchTableError`, 500/502/504 → `CommitStateUnknownException`.
- **Files:** on a definite failure (409, 404) the files written for the
transactions are deleted. On an unknown outcome they're kept, because the
catalog may already reference them. That's the same rule `commit_transaction`
follows. A failed transaction can't be reused.
- **Rejected before anything is sent:** no transactions, the same
transaction twice, two transactions on one table, a transaction from another
catalog, and `CreateTableTransaction`. Transactions without staged changes are
skipped, and a call with only empty ones sends nothing.
- **Endpoint check:** uses the server's advertised endpoints. For servers
that don't list endpoints, `transactions/commit` is assumed supported, as in
the Java client's `DEFAULT_ENDPOINTS`.
### Not in this PR (open questions)
- **No retry on conflict.** A 409 doesn't say which table conflicted, and
replaying snapshot producers across tables is a bigger design. The caller
re-stages and commits again, as with the Java client. Happy to follow up if
you'd like retries.
- **Empty transactions with snapshot producers** (for example a delete whose
filter matched nothing) are skipped, so their concurrency validation doesn't
run here. Should they be validated, or rejected?
- **Creating tables** in a multi-table commit (with `assert-create`) could
come later.
- **Entry point:** I went with `Catalog.commit_transactions(*transactions)`
as proposed in #4095. Glad to change it, for example to a context manager.
## Are these changes tested?
- `tests/catalog/test_rest.py`: 12 tests with a mocked server. They cover
the request body, the prefix, each error status, file cleanup only when nothing
was applied, skipping empty transactions, the input checks and an unsupported
endpoint.
- `tests/catalog/test_base.py`: the base catalog raises
`NotImplementedError`.
- `tests/integration/test_rest_catalog.py`, against the
`apache/iceberg-rest-fixture` with S3-compatible storage:
- two tables each get a real append (and a property change) in one call,
and both have the rows afterwards;
- another writer appends to one table after the transactions were staged:
the commit fails with `CommitFailedException`, and neither table gets the
staged rows.
`make test`: 4273 passed. `make lint` passes on the changed files.
## Are there any user-facing changes?
Yes: the new `Catalog.commit_transactions` method, and a "Multi-table
commits" section in `mkdocs/docs/api.md`.
I used an AI coding assistant while writing this change; I reviewed and
tested all of it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011ZBQVsXRmKT8WXDr2qSfbd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
