pvary commented on issue #10892: URL: https://github.com/apache/iceberg/issues/10892#issuecomment-2290980634
Discussed the issue with @stevenzwu yesterday in detail. We agreed, that we should separate out the problems above. 1. We need to prevent the sink to silently not commit data when there is already one committed to the table with the same checkpointId - this is only desired when restoring the state (as we can't be sure where the previous deployment failed). In every other case, when there are new incoming records, they need to be committed to the Iceberg table. 2. We need to find a way for the users to easily (manually or automatically) revert back changes done by the given job to the state what was created by the given checkpoint/savepoint. While in this specific case 1. and 2. was correlated, there might be cases when the desired behaviour is different, so we should not tie the 2 problems together. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
