antonlin1 opened a new pull request, #4162:
URL: https://github.com/apache/iceberg-python/pull/4162
# Rationale for this change
Replacing a dataset with an incompatible schema currently requires callers
to drop and recreate the table or assemble low-level metadata updates. Dropping
first can leave the table missing if the replacement write fails.
Add `Catalog.replace_table_transaction` to publish a replacement definition
and its data in one commit:
```python
with catalog.replace_table_transaction("db.table", schema=df.schema) as txn:
txn.append(df)
```
The implementation follows Java's [table metadata
replacement](https://github.com/apache/iceberg/blob/8c1ee9d9df4184a7889f85f6a9e1f82f959e0eaf/core/src/main/java/org/apache/iceberg/TableMetadata.java#L718)
and [REST replacement
transaction](https://github.com/apache/iceberg/blob/8c1ee9d9df4184a7889f85f6a9e1f82f959e0eaf/core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java#L1052).
It reuses column IDs by name and matching historical schema, partition-spec,
and sort-order IDs. It retains the table UUID, old snapshots, and other refs
while starting replacement data on a new snapshot lineage. The existing
transaction and catalog commit paths provide publication and retry handling.
Commit requirements protect ID allocation and concurrent metadata changes.
For v1 tables, removed partition fields become void fields. The change also
handles writes when their source columns have been removed, including Avro
encoding of null-only partition fields.
## Are these changes tested?
The full unit suite and repository lint pass. Behavioral tests cover
visibility before commit, incompatible schemas, historical reads, empty
replacements, exceptions, stale UUIDs, concurrent appends and schema changes,
ID allocation, layout replacement, and format upgrades. REST request tests
verify reconstruction through the normal commit endpoint. Null-field Avro tests
read the output with fastavro to check encoding independently.
The new integration test passes against the Apache Iceberg Java REST fixture
for both v1 and v2 tables. It verifies that the old rows remain readable until
commit, then checks the replacement rows and retained UUID, snapshots, and
tags. It also verifies a subsequent empty replacement.
## Are there any user-facing changes?
Yes. The new API accepts an Iceberg or Arrow schema and optional location,
partition spec, sort order, and properties. Partitioning and sorting default to
unpartitioned and unsorted. Properties merge with existing properties and
location defaults to the existing location. The table must already exist. An
exception in the context manager leaves the published table unchanged.
Concurrent schema or layout conflicts can require starting a new replacement
transaction.
The API documentation includes an example and these semantics. No new REST
endpoint or dependency is introduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
