github-actions[bot] commented on code in PR #68648:
URL: https://github.com/apache/doris/pull/68648#discussion_r4271997436
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescTransferGenerator.java:
##########
@@ -65,6 +75,38 @@ public void apply(MTMVPartitionInfo mvPartitionInfo,
Map<String, String> mvPrope
lastResult.setRes(res);
}
+ /**
+ * One MV partition per set of keys that meet, whichever table wrote them
down: a partition of a list
+ * partitioned base table can hold several keys of the MV's partition
column, so two partitions -- of one
+ * table or of two -- can describe keys that meet, and an MV's own
partitions cannot overlap. The one
+ * partition that holds a group's keys names every table's partitions of
them, which is what a refresh
+ * reads and records for those keys (see {@link
MTMVPartitionUtil#mergedListDescs}).
+ *
+ * <p>Merging across tables, not within each of them, is what keeps the MV
buildable: two tables of a
+ * multi-table MV have to come out with the same descs, or one table's
merged desc repeats a key another
+ * table's desc holds and `checkIntersect` (or the partition creation
itself) rejects the MV.
+ */
+ private Map<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>>
mergeOverlappingListDescs(
+ Map<MTMVRelatedTableIf, Map<PartitionKeyDesc, Set<String>>> descs)
{
+ Set<PartitionKeyDesc> allDescs = Sets.newHashSet();
+ for (Map<PartitionKeyDesc, Set<String>> onePctDescs : descs.values()) {
+ allDescs.addAll(onePctDescs.keySet());
+ }
+ Map<PartitionKeyDesc, PartitionKeyDesc> mergedOfDesc =
MTMVPartitionUtil.mergedListDescs(allDescs);
+ Map<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>> res =
Maps.newHashMap();
+ for (Entry<MTMVRelatedTableIf, Map<PartitionKeyDesc, Set<String>>>
entry : descs.entrySet()) {
+ for (Entry<PartitionKeyDesc, Set<String>> onePctEntry :
entry.getValue().entrySet()) {
+ PartitionKeyDesc desc =
mergedOfDesc.getOrDefault(onePctEntry.getKey(), onePctEntry.getKey());
Review Comment:
[P1] Scope external PCT reads when merging LIST components
Let D0=2020-01-01 and D1=2038-01-01. Use OLAP A `LIST(d,k)` with
`p_kept={(D0,1),(D1,1)}`, Paimon E `LIST(d)` with one populated partition per
date, and a non-IVM `PARTITION BY(d)` view of `SELECT a.d,a.k,e.v FROM A a JOIN
E e ON a.d=e.d` with `partition_sync_limit=2` and
`partition_sync_time_unit=YEAR`. The window retains A.p_kept and E[D1] only;
this merge newly admits M={D0,D1}, whereas CREATE previously rejected the
overlapping descriptors. `mappedBasePartitions` scopes only `OlapTable`, so the
refresh has this reduced plan:
```text
InsertOverwrite(M)
Join(a.d=e.d)
Filter((d,k) in A.p_kept) -> Scan A
Filter(d IN(D0,D1)) -> Scan E
```
It materializes E[D0] but records only E[D1]; PCT tables are excluded from
whole-table snapshot comparison. After a remote DROP of E[D0] and metadata
refresh, E[D1]'s `lastFileCreationTime` and the mapping are unchanged. M
therefore remains synchronized, and with external MV rewrite enabled the join
can read MV[M] and return the deleted D0 row. Scope external PCT reads to their
mapped partitions too, or record every external partition the MV-domain
predicate reads, and cover refresh followed by remote DROP. This extra external
read is distinct from the existing missing-OLAP-compensation finding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
