LuciferYang opened a new issue, #18677:
URL: https://github.com/apache/iceberg/issues/18677
### Apache Iceberg version
main (development)
### Query engine
None
### Please describe the bug 🐞
`TableMetadata.Builder.removeSchemas` filters `schemas` but leaves the
removed ids in `schemasById`, and `removeSpecs` does the same with `specsById`.
Later calls on the same builder still resolve the removed ids.
`CatalogHandlers.commit` applies every update in a request to one builder,
so a transaction that expires unused metadata and then changes the schema hits
this. Take a table whose current schema is 0 and whose schema 1 is unused
because a column was added and then dropped:
```java
table.updateSchema().addColumn("col_to_delete",
Types.IntegerType.get()).commit();
table.updateSchema().deleteColumn("col_to_delete").commit();
Transaction transaction = table.newTransaction();
transaction.expireSnapshots().cleanExpiredMetadata(true).commit();
transaction.updateSchema().addColumn("new_col",
Types.LongType.get()).commit();
transaction.commitTransaction();
```
The transaction sends `remove-schemas [1]`, then `add-schema` for a new
schema that the client assigned the freed id 1, then `set-current-schema -1`.
On the server, `addSchema` finds id 1 in the stale index and does not add the
new schema to `schemas`, and `setCurrentSchema(1)` picks up the removed schema
instead. `build()` passes, and the server commits metadata with
`current-schema-id: 1` and only schema 0. The `ops.current()` call at the end
of `CatalogHandlers.commit` then fails to read it back:
```
java.lang.IllegalArgumentException: Cannot find schema with
current-schema-id=1 from schemas
at
org.apache.iceberg.TableMetadataParser.fromJson(TableMetadataParser.java:386)
...
at
org.apache.iceberg.rest.CatalogHandlers.commit(CatalogHandlers.java:672)
```
The client sees a failed commit, but the metadata pointer has already moved,
so every later `loadTable` fails with the same error.
The partition spec version (add a partition field and remove it, then
`cleanExpiredMetadata(true)` and `updateSpec().addField(...)` in one
transaction) is rejected instead. The new spec is skipped the same way, and
`set-default-spec -1` fails with `Cannot set last added spec: no spec has been
added`.
`InMemoryCatalog` and `JdbcCatalog` are not affected, because
`BaseTransaction` applies each operation to freshly built metadata. REST
servers that commit through `CatalogHandlers`, or through their own copy of its
commit loop, are affected.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
