dragonknightk opened a new pull request, #4205: URL: https://github.com/apache/iceberg-python/pull/4205
Closes #4190 # Rationale for this change `UpdateSchema._commit` only sent `AssertCurrentSchemaId`, so a stale `update_schema` could commit after another writer had moved the current schema away and back. For example: - writer A adds a column (field id 2, schema 1), then drops it, which makes schema 0 current again - writer B, loaded before A's changes, adds a column: its `AssertCurrentSchemaId(0)` still passes, so it reuses field id 2 and adds a second schema with id 1 Java's `UpdateRequirements` adds `AssertLastAssignedFieldId(base.lastColumnId())` when a commit adds a schema (`update(MetadataUpdate.AddSchema)`). pyiceberg already defines `AssertLastAssignedFieldId` but never sent it. ### What changed - `UpdateSchema._commit` adds `AssertLastAssignedFieldId` with the transaction's `last_column_id` when it adds a new schema - this follows `UpdateSpec._commit`, which adds `AssertLastAssignedPartitionId` when it adds a spec - switching back to an existing schema adds no schema, so it only sends `AssertCurrentSchemaId`, as before With this change, writer B fails with `CommitFailedException` and the table stays unchanged (｀・ω・´) ## Are these changes tested? Yes. `test_concurrent_add_column_after_add_and_drop` in `tests/catalog/test_catalog_behaviors.py` runs the scenario from the issue against the memory and SQL catalogs: - with this change, writer B fails with `last assigned field id has changed: expected 3, found 4` - without it, the test fails with `DID NOT RAISE CommitFailedException` ## Are there any user-facing changes? Yes, a bug fix: a stale `update_schema` that adds a schema now raises `CommitFailedException` when the table's last assigned field id has changed, instead of reusing field ids. Refresh the table and retry, as with other requirement failures (・ω・)b ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change. All submitted content was reviewed by a human before submission. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
