aiborodin commented on code in PR #14810:
URL: https://github.com/apache/iceberg/pull/14810#discussion_r2622049013
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicCommitter.java:
##########
@@ -126,9 +119,15 @@ public void
commit(Collection<CommitRequest<DynamicCommittable>> commitRequests)
return;
}
- // For every table and every checkpoint, we store the list of
to-be-committed
- // DynamicCommittable.
- // There may be DynamicCommittable from previous checkpoints which have
not been committed yet.
+ /*
+ Each (table, branch, checkpoint) triplet must have only one commit
request.
+ There may be commit requests from previous checkpoints which have not
been committed yet.
+
+ We currently keep a List of commit requests per checkpoint instead of a
single CommitRequest<DynamicCommittable>
+ to process the Flink state from previous releases, which had multiple
commit requests due to a bug in the
+ upstream DynamicWriteResultAggregator. Iceberg 1.11.0 will remove this,
and users should upgrade to the latest
Review Comment:
Acknowledged
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicWriteResultAggregator.java:
##########
@@ -57,11 +58,11 @@ class DynamicWriteResultAggregator
implements OneInputStreamOperator<
CommittableMessage<DynamicWriteResult>,
CommittableMessage<DynamicCommittable>> {
private static final Logger LOG =
LoggerFactory.getLogger(DynamicWriteResultAggregator.class);
- private static final byte[] EMPTY_MANIFEST_DATA = new byte[0];
+ private static final byte[][] EMPTY_MANIFEST_DATA = new byte[0][];
Review Comment:
Good point. It appears that we don't need this because we use a Map in the
dynamic sink aggregator, which can't have an empty list of `WriteResults`. I
think it was ported from the `IcebergWriteAggregator`. I removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
