stevenzwu commented on code in PR #5642:
URL: https://github.com/apache/iceberg/pull/5642#discussion_r960193893
##########
flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergStreamWriter.java:
##########
@@ -83,9 +83,18 @@ public void dispose() throws Exception {
@Override
public void endInput() throws IOException {
// For bounded stream, it may don't enable the checkpoint mechanism so
we'd better to emit the
- // remaining
- // completed files to downstream before closing the writer so that we
won't miss any of them.
+ // remaining completed files to downstream before closing the writer so
that we won't miss any
+ // of them.
+ // Note that if the task is not closed after calling endInput, checkpoint
may be triggered again
+ // causing files to be sent repeatedly, so we need to set write to null
after the file is sent.
+
+ if (writer == null) {
+ return;
+ }
emit(writer.complete());
+
+ // Set write to null to prevent multiple calls to the complete method of
the same write
+ writer = null;
Review Comment:
Flink guarantees the single thread model btw `endInput` and
`prepareSnapshotPreBarrier`. agree with @xuzhiwen1255 that there should be no
race condition here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
