Guosmilesmile commented on PR #18303: URL: https://github.com/apache/iceberg/pull/18303#issuecomment-6433419138
@mxm Thanks for the suggestion. After thinking it over, I believe adding a new sinkId would be a better approach. Here’s what I’m currently code: * When checkpointing is enabled, I’ll generate the sinkId using a UUID. * In batch mode or when checkpointing is disabled, I’ll use the jobId as the sinkId. The sinkId will be stored in the state of the aggregation stage. When recovering from an existing state, the saved sinkId will be reused to avoid duplicate submissions. For batch mode, since checkpointing is not enabled, using the jobId as the sinkId should also help prevent duplicate data submissions when the job fails and is restarted. What do you think of this approach? Does it seem reasonable to you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
