bryanck commented on issue #6514: URL: https://github.com/apache/iceberg/issues/6514#issuecomment-2258607058
@mahdibh I also had the same thought. If we have an option or lower-level API that does not refresh, then the caller can handle failures and retries how it sees fit when the commit's base metadata does not match the current table state. This would allow more control by the caller, and we wouldn't need to pass down conditions. In the Kafka Connect sink, we could use this API or option to add more protections against having two coordinators active at the same time. For example, one coordinator loads a table to check offsets in the snapshot properties, and they match, so the commit proceeds. Concurrently a second coordinator completes a commit, so the metadata has changed. Now the first coordinator attempts a commit. `SnapshotProducer.apply()` currently will refresh the table and base the commit on the refreshed metadata, and this commit should succeed. If we could disable the refresh, and base the commit on the original loaded metadata, then this commit should fail (which is what we want). @fqaiser94 Did you have any thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
