Guosmilesmile commented on PR #18144: URL: https://github.com/apache/iceberg/pull/18144#issuecomment-5708857680
@swapna267 I agree that resolving the snapshot ID at Job Submission time works well when the cache is not refreshed, with an explicit failure if the snapshot has expired. However, with periodic refresh, a long-running job may encounter a failover after the original snapshot has expired. Resolving the snapshot ID at Job Submission would require manual intervention to restart the job, which doesn't seem ideal for the periodic refresh use case. I also noticed that other connectors supporting lookup joins generally establish their connections independently on each TaskManager. So I don't think resolving the snapshot ID at the Job Submission level is a good fit for this use case. This is also why I'm still leaning toward using the latest snapshot rather than pinning to a specific snapshot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
