lrsb commented on code in PR #16011:
URL: https://github.com/apache/iceberg/pull/16011#discussion_r3130380904
##########
flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicCommitter.java:
##########
@@ -119,43 +119,51 @@ public void
commit(Collection<CommitRequest<DynamicCommittable>> commitRequests)
DynamicWriteResultAggregator. Iceberg 1.12 will remove this, and users
should upgrade to the 1.11 release first
to migrate their state to a single commit request per checkpoint.
*/
- Map<TableKey, NavigableMap<Long, List<CommitRequest<DynamicCommittable>>>>
commitRequestMap =
- Maps.newHashMap();
+ Map<TableKey, Map<JobOperatorKey, NavigableMap<Long,
List<CommitRequest<DynamicCommittable>>>>>
+ commitRequestMap = Maps.newHashMap();
for (CommitRequest<DynamicCommittable> request : commitRequests) {
- NavigableMap<Long, List<CommitRequest<DynamicCommittable>>> committables
=
- commitRequestMap.computeIfAbsent(
- new TableKey(request.getCommittable()), unused ->
Maps.newTreeMap());
- committables
- .computeIfAbsent(request.getCommittable().checkpointId(), unused ->
Lists.newArrayList())
+ DynamicCommittable committable = request.getCommittable();
+ commitRequestMap
+ .computeIfAbsent(committable.key(), unused -> Maps.newHashMap())
+ .computeIfAbsent(new JobOperatorKey(committable), unused ->
Maps.newTreeMap())
+ .computeIfAbsent(committable.checkpointId(), unused ->
Lists.newArrayList())
.add(request);
}
- for (Map.Entry<TableKey, NavigableMap<Long,
List<CommitRequest<DynamicCommittable>>>> entry :
- commitRequestMap.entrySet()) {
- Table table =
catalog.loadTable(TableIdentifier.parse(entry.getKey().tableName()));
- DynamicCommittable last =
entry.getValue().lastEntry().getValue().get(0).getCommittable();
- Snapshot latestSnapshot = table.snapshot(entry.getKey().branch());
- Iterable<Snapshot> ancestors =
+ for (Map.Entry<
+ TableKey,
+ Map<JobOperatorKey, NavigableMap<Long,
List<CommitRequest<DynamicCommittable>>>>>
+ tableEntry : commitRequestMap.entrySet()) {
+ TableKey tableKey = tableEntry.getKey();
+ Table table =
catalog.loadTable(TableIdentifier.parse(tableKey.tableName()));
+ Snapshot latestSnapshot = table.snapshot(tableKey.branch());
+ List<Snapshot> ancestors =
Review Comment:
Because the iterator is lazy, not a big deal. Reverted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
