mxm commented on issue #17140: URL: https://github.com/apache/iceberg/issues/17140#issuecomment-4933189818
`StandardSinkTopologies.addGlobalCommitter` is a bit of a workaround because it doesn't replace the committer, but runs as post-commit. Under the hood, it creates an operator with parallelism 1. Unfortunately, this is only available for Flink 2.0 and above (we still support Flink 1.20). It could be a viable path. On the other hand, the current code works fine, apart from the catalog load issue. We already aggregate all the WriteResults and send them only to a committer subtask 0. So the rest of the committer subtasks never do anything. I would suggest to just check the subtask id in the committer and return for any subtasks id > 0. We should make sure to throw in case any messages ever make it to a subtask id > 0. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
