This is an automated email from the ASF dual-hosted git repository.
jacktengg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a1b1cf722fb [fix](fe) Enforce minimum buckets for automatic partition
fallbacks (#69353)
a1b1cf722fb is described below
commit a1b1cf722fba9526123214c574394c3d885dc973
Author: TengJianPing <[email protected]>
AuthorDate: Mon Oct 26 11:06:18 2026 +0800
[fix](fe) Enforce minimum buckets for automatic partition fallbacks (#69353)
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: Raising autobucket_min_buckets after creating an AUTO
RANGE partitioned table could leave new partitions below the minimum.
The shared calculator returned the stored default without applying the
current minimum when history was absent, empty, or unavailable. It also
reused a historical bucket count directly when size statistics had not
arrived.
Apply the current minimum to automatic default and historical bucket
counts in the shared calculator used by automatic range and dynamic
partitions. Keep fixed bucket counts and existing partitions unchanged.
Cover the fallback paths with unit tests and the configuration change
with a generated SQL regression.
In function AutoBucketCalculator.calculateAutoBuckets, calc.first can be
smaller than Config.autobucket_min_buckets. calculateBuckets() has a
branch that directly reuses the historical bucket count without applying
the current minimum.
In DynamicPartitionScheduler.java:273
(fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java:273):
if (hasDataPartitions.size() == sizeUnknownArray.size()) {
LOG.info("TabletStatMgr not synchronized partitions size yet, so use
previous partition bucket num");
return Pair.of(previousPartitionBucketsNum,
previousPartitionBucketsNum);
}
The returned candidate bucket count falls below the minimum when all
three conditions hold:
1. hasDataPartitions is nonempty. Partitions qualify as “having data”
based on visibleVersion >= 2, as shown in filterDataPartitions:242
(fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java:242).
2. Every selected historical partition has
getDataSizeExcludeEmptyReplica(true) <= 0. A typical case is that data
has been committed, but FE has not yet synchronized the size statistics.
Visible versions and size statistics are updated separately.
3. The last selected historical partition has fewer buckets than the
current autobucket_min_buckets.
For example:
- Initially, autobucket_min_buckets = 1, and a historical partition is
created with 1 bucket.
- A write succeeds, advancing its visible version to 2, while the size
statistics available to FE remain 0.
- autobucket_min_buckets is increased to 4.
- Creation of the next partition is triggered, with no selected
historical partition reporting a positive size.
- calculateBuckets() takes the branch above and returns Pair.of(1, 1),
so the candidate bucket count is 1 < 4.
- The added Math.max(calc.first, minBuckets) raises the candidate count
to 4, while previousBuckets retains the actual historical value of 1.
If at least one historical partition reports a positive size, the normal
estimation path calls:
AutoBucketUtils.getBucketsNum(uncompressedPartitionSize,
Config.autobucket_min_buckets)
That overload already applies Math.max(minBuckets, bucketsNum), as shown
in AutoBucketUtils.java:126
(fe/fe-core/src/main/java/org/apache/doris/common/util/AutoBucketUtils.java:126).
Assuming the configuration remains unchanged during calculation, this
path cannot produce a candidate count below the minimum.
The new logic covers the path that directly reuses the old bucket count
when all historical size statistics are unavailable.
AutoBucketCalculator reports this path as successful, so the earlier
minimum enforcement on defaultBuckets does not cover it.
The commit’s testUnknownHistoricalSizesUsesCurrentMinimum:119
(fe/fe-core/src/test/java/org/apache/doris/common/util/AutoBucketCalculatorTest.java:119)
covers this case with visible version 2, size 0, historical bucket
counts of 1, 2, and 8, and a current minimum of 4.
### Release note
New automatic range and dynamic partitions respect
autobucket_min_buckets when reusing defaults or historical bucket counts
without size statistics. Existing partitions and fixed bucket
distributions retain their bucket counts.
### Check List (For Author)
- Test: Unit Test / Regression test / FE build and Checkstyle
- run-fe-ut.sh: AutoBucketCalculatorTest and
DynamicPartitionTableTest#testAutoBuckets; 13 tests passed
- run-regression-test.sh: test_autobucket_min_buckets,
test_autobucket_dynamic_partition, test_modify_distribution_buckets; 3
suites passed
- Generated the new .out through run-regression-test.sh -genOut and
verified it in comparison mode
- Built FE with build.sh --fe and passed mvn checkstyle:check -pl
fe-core
- SQL validation reused the worktree BE with be_exec_version=17 in the
isolated cluster
- Behavior changed: Yes (new automatically bucketed partitions enforce
the current minimum on fallback paths)
- Does this need documentation: No (restore the existing minimum-bucket
configuration semantics)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
.../doris/common/util/AutoBucketCalculator.java | 12 +-
.../common/util/AutoBucketCalculatorTest.java | 139 +++++++++++++++++++++
.../autobucket/test_autobucket_min_buckets.out | 11 ++
.../autobucket/test_autobucket_min_buckets.groovy | 109 ++++++++++++++++
4 files changed, 268 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/AutoBucketCalculator.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/AutoBucketCalculator.java
index 0576712f083..cd01733f417 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/util/AutoBucketCalculator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/util/AutoBucketCalculator.java
@@ -20,6 +20,7 @@ package org.apache.doris.common.util;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Partition;
import org.apache.doris.clone.DynamicPartitionScheduler;
+import org.apache.doris.common.Config;
import org.apache.doris.common.Pair;
import org.apache.doris.rpc.RpcException;
@@ -130,10 +131,14 @@ public class AutoBucketCalculator {
String partitionName = context.getPartitionName();
String nowPartitionName = context.getNowPartitionName();
boolean executeFirstTime = context.isExecuteFirstTime();
- int defaultBuckets = context.getDefaultBuckets();
+ boolean autoBucket = table.isAutoBucket();
+ int minBuckets = Config.autobucket_min_buckets;
+ // Stored defaults may predate a change to the minimum. Fixed-bucket
tables keep their configured count.
+ int defaultBuckets = autoBucket
+ ? Math.max(context.getDefaultBuckets(), minBuckets) :
context.getDefaultBuckets();
// if execute first time or not auto bucket, use default buckets
- if (!table.isAutoBucket() || executeFirstTime) {
+ if (!autoBucket || executeFirstTime) {
return AutoBucketResult.fallback(defaultBuckets,
executeFirstTime ? "executeFirstTime" : "not auto bucket
table");
}
@@ -174,7 +179,8 @@ public class AutoBucketCalculator {
// Calculate buckets based on historical data
Pair<Integer, Integer> calc =
DynamicPartitionScheduler.calculateBuckets(hasDataPartitions);
- int candidateBuckets = calc.first;
+ // Historical bucket counts may also predate a change to the minimum
when size statistics are unavailable.
+ int candidateBuckets = Math.max(calc.first, minBuckets);
int previousBuckets = calc.second;
return AutoBucketResult.success(candidateBuckets, previousBuckets);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/util/AutoBucketCalculatorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/util/AutoBucketCalculatorTest.java
index 4b23a1f0d13..015b25918ec 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/common/util/AutoBucketCalculatorTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/common/util/AutoBucketCalculatorTest.java
@@ -17,17 +17,156 @@
package org.apache.doris.common.util;
+import org.apache.doris.catalog.HashDistributionInfo;
import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.RangePartitionInfo;
+import org.apache.doris.clone.DynamicPartitionScheduler;
+import org.apache.doris.common.Config;
import org.apache.doris.common.util.AutoBucketCalculator.AutoBucketContext;
import org.apache.doris.common.util.AutoBucketCalculator.AutoBucketResult;
+import org.apache.doris.rpc.RpcException;
+import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.mockito.MockedStatic;
import org.mockito.Mockito;
+import java.util.Collections;
+import java.util.List;
+
public class AutoBucketCalculatorTest {
OlapTable table = Mockito.mock(OlapTable.class);
+ private int originalMinBuckets;
+
+ @BeforeEach
+ public void setUp() {
+ originalMinBuckets = Config.autobucket_min_buckets;
+ Config.autobucket_min_buckets = 4;
+ RangePartitionInfo partitionInfo =
Mockito.mock(RangePartitionInfo.class);
+
Mockito.when(partitionInfo.getIdToItem(false)).thenReturn(Collections.emptyMap());
+ Mockito.when(table.getPartitionInfo()).thenReturn(partitionInfo);
+ }
+
+ @AfterEach
+ public void tearDown() {
+ Config.autobucket_min_buckets = originalMinBuckets;
+ }
+
+ @Test
+ public void testFixedBucketsBelowMinimum() {
+ for (int buckets : new int[] {1, 2}) {
+ AutoBucketContext context = new AutoBucketContext(table, "p1",
"p2", false, buckets);
+ Assertions.assertEquals(buckets,
AutoBucketCalculator.calculateAutoBuckets(context).getBuckets());
+ Assertions.assertEquals(buckets,
AutoBucketCalculator.calculateAutoBucketsWithBoundsCheck(context));
+ }
+ }
+
+ @Test
+ public void testFirstExecutionUsesCurrentMinimum() {
+ Mockito.when(table.isAutoBucket()).thenReturn(true);
+ for (int buckets : new int[] {1, 2, 8}) {
+ AutoBucketContext context = new AutoBucketContext(table, "p1",
"p2", true, buckets);
+ Assertions.assertEquals(Math.max(4, buckets),
+
AutoBucketCalculator.calculateAutoBuckets(context).getBuckets());
+ Assertions.assertEquals(Math.max(4, buckets),
+
AutoBucketCalculator.calculateAutoBucketsWithBoundsCheck(context));
+ }
+ }
+
+ @Test
+ public void testNoHistoricalPartitionsUsesCurrentMinimum() throws
RpcException {
+ assertFallbackUsesCurrentMinimum(Collections.emptyList(),
Collections.emptyList(), "no data partitions");
+ }
+
+ @Test
+ public void testEmptyHistoricalPartitionUsesCurrentMinimum() throws
RpcException {
+
assertFallbackUsesCurrentMinimum(Collections.singletonList(Mockito.mock(Partition.class)),
+ Collections.singletonList(1L), "no data partitions");
+ }
+
+ @Test
+ public void testVisibleVersionsMismatchUsesCurrentMinimum() throws
RpcException {
+
assertFallbackUsesCurrentMinimum(Collections.singletonList(Mockito.mock(Partition.class)),
+ Collections.emptyList(), "visible versions mismatch");
+ }
+
+ @Test
+ public void testMissingVisibleVersionsUsesCurrentMinimum() throws
RpcException {
+
assertFallbackUsesCurrentMinimum(Collections.singletonList(Mockito.mock(Partition.class)),
+ null, "visible versions mismatch");
+ }
+
+ @Test
+ public void testVisibleVersionRpcFailureUsesCurrentMinimum() throws
RpcException {
+ Mockito.when(table.isAutoBucket()).thenReturn(true);
+ List<Partition> partitions =
Collections.singletonList(Mockito.mock(Partition.class));
+ try (MockedStatic<DynamicPartitionScheduler> scheduler =
Mockito.mockStatic(
+ DynamicPartitionScheduler.class,
Mockito.CALLS_REAL_METHODS);
+ MockedStatic<Partition> partitionMethods =
Mockito.mockStatic(Partition.class)) {
+ scheduler.when(() ->
DynamicPartitionScheduler.getHistoricalPartitions(table, "p2"))
+ .thenReturn(partitions);
+ partitionMethods.when(() ->
Partition.getVisibleVersions(partitions))
+ .thenThrow(new RpcException("", "test failure"));
+ assertFallbackBuckets("RpcException: test failure");
+ }
+ }
+
+ @Test
+ public void testUnknownHistoricalSizesUsesCurrentMinimum() throws
RpcException {
+ Mockito.when(table.isAutoBucket()).thenReturn(true);
+ Partition partition = Mockito.mock(Partition.class);
+ HashDistributionInfo distribution =
Mockito.mock(HashDistributionInfo.class);
+ Mockito.when(partition.getDistributionInfo()).thenReturn(distribution);
+
Mockito.when(partition.getDataSizeExcludeEmptyReplica(true)).thenReturn(0L);
+ List<Partition> partitions = Collections.singletonList(partition);
+ try (MockedStatic<DynamicPartitionScheduler> scheduler =
Mockito.mockStatic(
+ DynamicPartitionScheduler.class,
Mockito.CALLS_REAL_METHODS);
+ MockedStatic<Partition> partitionMethods =
Mockito.mockStatic(Partition.class)) {
+ scheduler.when(() ->
DynamicPartitionScheduler.getHistoricalPartitions(table, "p2"))
+ .thenReturn(partitions);
+ partitionMethods.when(() ->
Partition.getVisibleVersions(partitions))
+ .thenReturn(Collections.singletonList(2L));
+ for (int buckets : new int[] {1, 2, 8}) {
+ Mockito.when(distribution.getBucketNum()).thenReturn(buckets);
+ AutoBucketContext context = new AutoBucketContext(table, "p1",
"p2", false, 1);
+ AutoBucketResult result =
AutoBucketCalculator.calculateAutoBuckets(context);
+ Assertions.assertTrue(result.isSuccess());
+ Assertions.assertEquals(Math.max(4, buckets),
result.getBuckets());
+ Assertions.assertEquals(buckets, result.getPreviousBuckets());
+ Assertions.assertEquals(Math.max(4, buckets),
+
AutoBucketCalculator.calculateAutoBucketsWithBoundsCheck(context));
+ }
+ }
+ }
+
+ private void assertFallbackUsesCurrentMinimum(List<Partition> partitions,
List<Long> versions, String reason)
+ throws RpcException {
+ Mockito.when(table.isAutoBucket()).thenReturn(true);
+ try (MockedStatic<DynamicPartitionScheduler> scheduler =
Mockito.mockStatic(
+ DynamicPartitionScheduler.class,
Mockito.CALLS_REAL_METHODS);
+ MockedStatic<Partition> partitionMethods =
Mockito.mockStatic(Partition.class)) {
+ scheduler.when(() ->
DynamicPartitionScheduler.getHistoricalPartitions(table, "p2"))
+ .thenReturn(partitions);
+ partitionMethods.when(() ->
Partition.getVisibleVersions(partitions)).thenReturn(versions);
+ assertFallbackBuckets(reason);
+ }
+ }
+
+ private void assertFallbackBuckets(String reason) {
+ for (int buckets : new int[] {1, 2, 8}) {
+ AutoBucketContext context = new AutoBucketContext(table, "p1",
"p2", false, buckets);
+ AutoBucketResult result =
AutoBucketCalculator.calculateAutoBuckets(context);
+ Assertions.assertFalse(result.isSuccess());
+ Assertions.assertEquals(reason, result.getErrorMessage());
+ Assertions.assertEquals(Math.max(4, buckets), result.getBuckets());
+ Assertions.assertEquals(Math.max(4, buckets),
+
AutoBucketCalculator.calculateAutoBucketsWithBoundsCheck(context));
+ }
+ }
@Test
public void testCalculateAutoBucketsNotAutoBucket() {
diff --git a/regression-test/data/autobucket/test_autobucket_min_buckets.out
b/regression-test/data/autobucket/test_autobucket_min_buckets.out
new file mode 100644
index 00000000000..5be8e8a659c
--- /dev/null
+++ b/regression-test/data/autobucket/test_autobucket_min_buckets.out
@@ -0,0 +1,11 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !buckets --
+autobucket_min_fixed p20261021000000 2
+autobucket_min_history p20261020000000 1
+autobucket_min_history p20261021000000 4
+autobucket_min_one p20261021000000 4
+autobucket_min_two p20261021000000 4
+
+-- !dynamic_buckets --
+4
+
diff --git
a/regression-test/suites/autobucket/test_autobucket_min_buckets.groovy
b/regression-test/suites/autobucket/test_autobucket_min_buckets.groovy
new file mode 100644
index 00000000000..b95ef4b5002
--- /dev/null
+++ b/regression-test/suites/autobucket/test_autobucket_min_buckets.groovy
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_autobucket_min_buckets", "nonConcurrent") {
+ setFeConfigTemporary([
+ "autobucket_min_buckets": "1",
+ "autobucket_partition_size_per_bucket_gb": "5"
+ ]) {
+ sql "DROP TABLE IF EXISTS autobucket_min_one"
+ sql """
+ CREATE TABLE autobucket_min_one (
+ dt DATETIMEV2(0) NOT NULL,
+ logid BIGINT NOT NULL,
+ content VARIANT NULL
+ ) UNIQUE KEY(dt, logid)
+ AUTO PARTITION BY RANGE(date_trunc(dt, 'day')) ()
+ DISTRIBUTED BY HASH(logid) BUCKETS AUTO
+ PROPERTIES (
+ "replication_num" = "1",
+ "estimate_partition_size" = "1M",
+ "enable_unique_key_merge_on_write" = "true"
+ )
+ """
+
+ sql "DROP TABLE IF EXISTS autobucket_min_two"
+ sql """
+ CREATE TABLE autobucket_min_two (
+ dt DATETIMEV2(0) NOT NULL,
+ logid BIGINT NOT NULL,
+ content VARIANT NULL
+ ) UNIQUE KEY(dt, logid)
+ AUTO PARTITION BY RANGE(date_trunc(dt, 'day')) ()
+ DISTRIBUTED BY HASH(logid) BUCKETS AUTO
+ PROPERTIES (
+ "replication_num" = "1",
+ "estimate_partition_size" = "30G",
+ "enable_unique_key_merge_on_write" = "true"
+ )
+ """
+
+ sql "DROP TABLE IF EXISTS autobucket_min_history"
+ sql "CREATE TABLE autobucket_min_history LIKE autobucket_min_one"
+ sql """INSERT INTO autobucket_min_history VALUES ('2026-10-20', 1,
CAST('{"v":1}' AS VARIANT))"""
+
+ sql "DROP TABLE IF EXISTS autobucket_min_fixed"
+ sql """
+ CREATE TABLE autobucket_min_fixed (
+ dt DATETIMEV2(0) NOT NULL,
+ logid BIGINT NOT NULL
+ ) DUPLICATE KEY(dt, logid)
+ AUTO PARTITION BY RANGE(date_trunc(dt, 'day')) ()
+ DISTRIBUTED BY HASH(logid) BUCKETS 2
+ PROPERTIES ("replication_num" = "1")
+ """
+
+ // Defaults and the existing one-bucket partition were created before
raising the minimum.
+ setFeConfig("autobucket_min_buckets", "4")
+ sql """INSERT INTO autobucket_min_one VALUES ('2026-10-21', 1,
CAST('{"v":1}' AS VARIANT))"""
+ sql """INSERT INTO autobucket_min_two VALUES ('2026-10-21', 1,
CAST('{"v":1}' AS VARIANT))"""
+ sql """INSERT INTO autobucket_min_history VALUES ('2026-10-21', 2,
CAST('{"v":2}' AS VARIANT))"""
+ sql "INSERT INTO autobucket_min_fixed VALUES ('2026-10-21', 1)"
+
+ // New AUTO partitions respect the new minimum; existing and
fixed-bucket partitions retain their counts.
+ order_qt_buckets """
+ SELECT TABLE_NAME, PARTITION_NAME, BUCKET_NUM
+ FROM information_schema.partitions
+ WHERE TABLE_SCHEMA = DATABASE()
+ AND TABLE_NAME IN ('autobucket_min_one', 'autobucket_min_two',
+ 'autobucket_min_history', 'autobucket_min_fixed')
+ """
+
+ // The dynamic partition scheduler uses the same minimum, including
its first execution.
+ sql "DROP TABLE IF EXISTS autobucket_min_dynamic"
+ sql """
+ CREATE TABLE autobucket_min_dynamic (
+ dt DATEV2 NOT NULL,
+ logid BIGINT NOT NULL
+ ) DUPLICATE KEY(dt, logid)
+ PARTITION BY RANGE(dt) ()
+ DISTRIBUTED BY HASH(logid) BUCKETS AUTO
+ PROPERTIES (
+ "replication_num" = "1",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.time_unit" = "DAY",
+ "dynamic_partition.end" = "1",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "1"
+ )
+ """
+ order_qt_dynamic_buckets """
+ SELECT DISTINCT BUCKET_NUM FROM information_schema.partitions
+ WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME =
'autobucket_min_dynamic'
+ """
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
