mudit-97 commented on PR #9694: URL: https://github.com/apache/iceberg/pull/9694#issuecomment-1968656628
@pvary we felt that even single phase commit should not have much of an issue if users want to use that Why? 1. Current implementation is 2 step so expectation with users is that once checkpoint's notifyCheckpointComplete is called then only data will be discoverable in Iceberg tables, but we felt that users should not have much of an issue if we do commit in snapshot state itself 2. Since this is a sink operator, ideally there will not be anyone depending on this operator and this will be finally writing the files only so it is better to do commit in single go 3. If we do single phase commit, there is no need to store checkpoints in external store also, we can ensure data will be committed before checkpoint is marked as complete 4. We will never face any issues of data loss if checkpoint is corrupted etc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
