LuciferYang opened a new pull request, #18678: URL: https://github.com/apache/iceberg/pull/18678
Closes #18677. `TableMetadata.Builder.removeSchemas` and `removeSpecs` filter `schemas` and `specs` but leave the removed ids in `schemasById` and `specsById`. `CatalogHandlers.commit` applies a whole request to one builder, so a transaction that runs `expireSnapshots().cleanExpiredMetadata(true)` and then adds a column or a partition field can get a freed id and run into the stale entry. For schemas, the server commits metadata whose `current-schema-id` is not in `schemas`, and the table can no longer be loaded. For specs, the commit is rejected with `Cannot set last added spec: no spec has been added`. This removes the ids from the indexes together with the list entries. `setCurrentSchema` on a removed id now fails with `Cannot set current schema to unknown schema`, and a new schema or spec that gets a freed id is added normally. The same bug was fixed in iceberg-cpp in apache/iceberg-cpp#1012. New tests: - `CatalogTests.testRemoveUnusedSchemaThenAddColumnInTransaction` and `testRemoveUnusedSpecThenAddFieldInTransaction` run that transaction against each catalog. Without the fix both fail on `TestRESTCatalog`; they pass on the in-memory, JDBC and Nessie catalogs either way. - `TestTableMetadata.testRemoveSchemasThenReuseRemovedId` and `testRemoveSpecsThenReuseRemovedId` run the same sequence on one builder, so the bug shows without a REST server. Without the fix, `setCurrentSchema(1)` after `removeSchemas([1])` succeeds, and the new spec is not added to `specs`. --- **AI Disclosure** - Model: Claude Opus - Human Oversight: reviewed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
