This is an automated email from the ASF dual-hosted git repository.
yujun777 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f6053753fa0 [fix](fe) Use concurrent map for OlapTable partition names
to avoid ConcurrentModificationException (#69166)
f6053753fa0 is described below
commit f6053753fa0739fc715db143fb5b483991848429
Author: yujun <[email protected]>
AuthorDate: Fri Oct 16 17:19:02 2026 +0800
[fix](fe) Use concurrent map for OlapTable partition names to avoid
ConcurrentModificationException (#69166)
`OlapTable.getPartitionNames()` copies `nameToPartition.keySet()`
(`Sets.newHashSet(...)`) without holding any metadata lock, while
`nameToPartition` was a plain (fail-fast) `TreeMap` that is mutated in
place by partition add/drop/rename under the table write lock. Any
lock-free reader therefore races with the writer and can throw
`java.util.ConcurrentModificationException`.
Reproduced on P0 regression
`nereids_rules_p0/mv/create_part_and_up/range_datetime_part_up_rewrite`:
the suite runs `refresh MATERIALIZED VIEW <mv> auto` (whose refresh job
adds partitions to the MV base table) and immediately `explain
materialized view rewrite` on a query that is rewritten onto that MV.
Rendering the memo plan calls `PhysicalOlapScan.toString()` ->
`OlapTable.getPartitionNames()`, and the concurrent partition addition
of the still-running refresh job makes the copy throw:
```
2026-10-16 01:42:05,315 INFO (mtmv-task-execute-1-thread-2)
[EditLog.logAddPartition():1825] log add partition, ... "tableId":1792084617789
...
2026-10-16 01:42:05,315 INFO (mysql-nio-pool-112)
[StmtExecutor.executeByNereids():1003] Command(explain memo plan ...) process
failed.
java.util.ConcurrentModificationException
at java.util.TreeMap$KeyIterator.next(TreeMap.java:1540)
at com.google.common.collect.Sets.newHashSet(Sets.java:226)
at
org.apache.doris.catalog.OlapTable.getPartitionNames(OlapTable.java:1643)
at
org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan.toString(PhysicalOlapScan.java:326)
at org.apache.doris.nereids.memo.Memo.toString(Memo.java:922)
at
org.apache.doris.nereids.NereidsPlanner.getExplainString(NereidsPlanner.java:1092)
at
org.apache.doris.nereids.trees.plans.commands.ExplainCommand.run(ExplainCommand.java:138)
```
The rewrite itself succeeded; only the EXPLAIN output rendering crashed,
so this shows up as a flaky failure. But `getPartitionNames()` is
reached from ~124 call sites (including MTMV partition sync checks that
run without a table lock), so this is a general latent race, not an
EXPLAIN-only one.
---
.../java/org/apache/doris/catalog/OlapTable.java | 7 +++++--
.../org/apache/doris/catalog/OlapTableTest.java | 21 +++++++++++++++++++++
2 files changed, 26 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index 8bd54777222..f1ef42ae1f4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -124,6 +124,7 @@ import java.util.Set;
import java.util.TreeMap;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
@@ -191,9 +192,11 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf, GsonPostProc
// to invalidate stale partition pruning metadata without reacting to
ordinary data writes. This is transient and
// rebuilt from zero after deserialization because prepared statement
cache is also in-memory.
private transient AtomicLong partitionTopologyVersion = new AtomicLong(0L);
- // handled in postgsonprocess
+ // handled in postgsonprocess. Sorted, and weakly consistent so that
lock-free readers
+ // (e.g. EXPLAIN plan rendering) never hit ConcurrentModificationException
while the table
+ // write lock holder adds/drops partitions.
@Getter
- protected Map<String, Partition> nameToPartition = Maps.newTreeMap();
+ protected Map<String, Partition> nameToPartition = new
ConcurrentSkipListMap<>();
@SerializedName(value = "di", alternate = {"distributionInfo"})
private DistributionInfo defaultDistributionInfo;
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java
index 2671b51b379..8890d83105b 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java
@@ -49,6 +49,7 @@ import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -185,6 +186,26 @@ public class OlapTableTest {
Assertions.assertEquals(version + 1,
olapTable.getPartitionTopologyVersion());
}
+ @Test
+ public void testPartitionNameMapToleratesConcurrentAdd() {
+ OlapTable olapTable = new OlapTable();
+ for (int i = 0; i < 3; i++) {
+ olapTable.addPartition(newPartitionForTopologyVersionTest(10 + i,
"p" + i));
+ }
+
+ // getPartitionNames() copies nameToPartition's keys without holding
any lock, while a writer
+ // holding the table write lock may add a partition at the same time,
so the iteration must be
+ // weakly consistent instead of fail fast.
+ Iterator<String> iterator =
olapTable.getNameToPartition().keySet().iterator();
+ Assertions.assertTrue(iterator.hasNext());
+ iterator.next();
+ olapTable.addPartition(newPartitionForTopologyVersionTest(20, "p10"));
+ while (iterator.hasNext()) {
+ iterator.next();
+ }
+ Assertions.assertEquals(4, olapTable.getPartitionNames().size());
+ }
+
private void addPartitionForTopologyVersionTest(OlapTable olapTable, long
partitionId, String partitionName) {
olapTable.getPartitionInfo().addPartition(partitionId, new
DataProperty(TStorageMedium.HDD),
new ReplicaAllocation((short) 1), false, true);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
