sourajitsaha17 opened a new issue, #14093:
URL: https://github.com/apache/iceberg/issues/14093
### Apache Iceberg version
1.9.2
### Query engine
Dremio
### Please describe the bug 🐞
Currently, the snapshot producer refreshes table metadata only once at the
beginning of the operation and then constructs new metadata from that. This
allows stale schema information to slip through if concurrent schema evolutions
occur after the initial refresh.
In our functional test, the steps are as follows:
* A schema evolution is detected by the application (e.g. adding a new
column), and the table’s schema is updated and committed successfully.
* Then on the next step the data file commit starts with the current metadata
* Another schema change happens before the data file commit finishes.
* The data file commit finalizes with outdated metadata, reverting the new
schema.
Note:
On adding some additional logging in the SanpshotProducer, we could confirm
it.
```
org.apache.iceberg.SnapshotProducer - Current base schema id: 5, updated
schema id: 2 {}
```
This log was added right before the commit method `taskOps.commit` from
SnapshotProducer. Performed a `table.refresh()` and comapred the schema id.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
