This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push: new 74bfd69 [Bug] Forbidden creating table with dynamic partition when FE.config dynamic_partition_enable=false (#5043) 74bfd69 is described below commit 74bfd695957c23d7a1bfec1b926dbd0c4997b564 Author: EmmyMiao87 <522274...@qq.com> AuthorDate: Wed Dec 16 23:44:20 2020 +0800 [Bug] Forbidden creating table with dynamic partition when FE.config dynamic_partition_enable=false (#5043) - There is a fe configuration called dynamic_partition_enable which controls the opening and closing of the dynamic partition function. When this configuration is false, it means that all tables do not support dynamic partitioning. - But when the user tried to create the dynamic partition table, Doris did not detect this parameter. This will cause the user can normally create a dynamic partition table, but in fact Doris cannot create a partition for this table. - This pr detect this config when building the table. The dynamic partition table can be created only when the dynamic_partition_enable configuration is true. If the configuration is false, the command to create a dynamic partition table will directly report an error. --- docs/en/administrator-guide/dynamic-partition.md | 10 +++- .../sql-statements/Data Definition/CREATE TABLE.md | 5 +- .../zh-CN/administrator-guide/dynamic-partition.md | 10 +++- .../sql-statements/Data Definition/CREATE TABLE.md | 5 +- .../java/org/apache/doris/catalog/Catalog.java | 6 +- .../org/apache/doris/catalog/TableProperty.java | 22 +++++-- .../doris/common/util/DynamicPartitionUtil.java | 68 +++++++++++++--------- .../common/util/DynamicPartitionUtilTest.java | 24 ++++++++ 8 files changed, 110 insertions(+), 40 deletions(-) diff --git a/docs/en/administrator-guide/dynamic-partition.md b/docs/en/administrator-guide/dynamic-partition.md index d92126a..ab23f44 100644 --- a/docs/en/administrator-guide/dynamic-partition.md +++ b/docs/en/administrator-guide/dynamic-partition.md @@ -83,7 +83,7 @@ The rules of dynamic partition are prefixed with `dynamic_partition.`: The unit for dynamic partition scheduling. Can be specified as `HOUR`,`DAY`,` WEEK`, and `MONTH`, means to create or delete partitions by hour, day, week, and month, respectively. - When specified as `HOUR`, the suffix format of the dynamically created partition name is `yyyyMMddHH`, for example, `2020032501`. + When specified as `HOUR`, the suffix format of the dynamically created partition name is `yyyyMMddHH`, for example, `2020032501`. *When the time unit is HOUR, the data type of partition column cannot be DATE.* When specified as `DAY`, the suffix format of the dynamically created partition name is `yyyyMMdd`, for example, `20200325`. @@ -341,3 +341,11 @@ When dynamic partitioning feature is enabled, Doris will no longer allow users t The dynamic partitioning feature can be disabled by executing `ALTER TABLE tbl_name SET ("dynamic_partition.enable" = "false") ` and converting it to a manual partition table. When dynamic partitioning feature is disabled, Doris will no longer manage partitions automatically, and users will have to create or delete partitions manually by using `ALTER TABLE`. + +## Common problem + +1. After creating the dynamic partition table, it prompts ```Could not create table with dynamic partition when fe config dynamic_partition_enable is false``` + + Because the main switch of dynamic partition, that is, the configuration of FE ```dynamic_partition_enable``` is false, the dynamic partition table cannot be created. + + At this time, please modify the FE configuration file, add a line ```dynamic_partition_enable=true```, and restart FE. Or execute the command ADMIN SET FRONTEND CONFIG ("dynamic_partition_enable" = "true") to turn on the dynamic partition switch. diff --git a/docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md b/docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md index 7213040..c984a7c 100644 --- a/docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md +++ b/docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md @@ -253,7 +253,7 @@ Syntax: ``` PROPERTIES ( "dynamic_partition.enable" = "true|false", - "dynamic_partition.time_unit" = "DAY|WEEK|MONTH", + "dynamic_partition.time_unit" = "HOUR|DAY|WEEK|MONTH", "dynamic_partitoin.end" = "${integer_value}", "dynamic_partition.prefix" = "${string_value}", "dynamic_partition.buckets" = "${integer_value} @@ -262,7 +262,8 @@ Syntax: Dynamic_partition. Enable: specifies whether dynamic partitioning at the table level is enabled - Dynamic_partition. Time_unit: used to specify the time unit for dynamically adding partitions, which can be selected as DAY, WEEK, and MONTH. + Dynamic_partition. Time_unit: used to specify the time unit for dynamically adding partitions, which can be selected as HOUR, DAY, WEEK, and MONTH. + Attention: When the time unit is HOUR, the data type of partition column cannot be DATE. Dynamic_partition. End: used to specify the number of partitions created in advance diff --git a/docs/zh-CN/administrator-guide/dynamic-partition.md b/docs/zh-CN/administrator-guide/dynamic-partition.md index c7a9237..5a83b02 100644 --- a/docs/zh-CN/administrator-guide/dynamic-partition.md +++ b/docs/zh-CN/administrator-guide/dynamic-partition.md @@ -81,7 +81,7 @@ under the License. 动态分区调度的单位。可指定为 `HOUR`、`DAY`、`WEEK`、`MONTH`。分别表示按天、按星期、按月进行分区创建或删除。 - 当指定为 `HOUR` 时,动态创建的分区名后缀格式为 `yyyyMMddHH`,例如`2020032501`。 + 当指定为 `HOUR` 时,动态创建的分区名后缀格式为 `yyyyMMddHH`,例如`2020032501`。小时为单位的分区列数据类型不能为 DATE。 当指定为 `DAY` 时,动态创建的分区名后缀格式为 `yyyyMMdd`,例如`20200325`。 @@ -339,3 +339,11 @@ mysql> SHOW DYNAMIC PARTITION TABLES; 通过执行 `ALTER TABLE tbl_name SET ("dynamic_partition.enable" = "false")` 即可关闭动态分区功能,将其转换为手动分区表。 关闭动态分区功能后,Doris 将不再自动管理分区,需要用户手动通过 `ALTER TABLE` 的方式创建或删除分区。 + +## 常见问题 + +1. 创建动态分区表后提示 ```Could not create table with dynamic partition when fe config dynamic_partition_enable is false``` + + 由于动态分区的总开关,也就是 FE 的配置 ```dynamic_partition_enable``` 为 false,导致无法创建动态分区表。 + + 这时候请修改 FE 的配置文件,增加一行 ```dynamic_partition_enable=true```,并重启 FE。或者执行命令 ADMIN SET FRONTEND CONFIG ("dynamic_partition_enable" = "true") 将动态分区开关打开即可。 diff --git a/docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE TABLE.md b/docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE TABLE.md index 3532032..7186552 100644 --- a/docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE TABLE.md +++ b/docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE TABLE.md @@ -276,14 +276,15 @@ under the License. ``` PROPERTIES ( "dynamic_partition.enable" = "true|false", - "dynamic_partition.time_unit" = "DAY|WEEK|MONTH", + "dynamic_partition.time_unit" = "HOUR|DAY|WEEK|MONTH", "dynamic_partition.start" = "${integer_value}", "dynamic_partitoin.end" = "${integer_value}", "dynamic_partition.prefix" = "${string_value}", "dynamic_partition.buckets" = "${integer_value} ``` dynamic_partition.enable: 用于指定表级别的动态分区功能是否开启。默认为 true。 - dynamic_partition.time_unit: 用于指定动态添加分区的时间单位,可选择为DAY(天),WEEK(周),MONTH(月) + dynamic_partition.time_unit: 用于指定动态添加分区的时间单位,可选择为HOUR(小时),DAY(天),WEEK(周),MONTH(月)。 + 注意:以小时为单位的分区列,数据类型不能为 DATE。 dynamic_partition.start: 用于指定向前删除多少个分区。值必须小于0。默认为 Integer.MIN_VALUE。 dynamic_partition.end: 用于指定提前创建的分区数量。值必须大于0。 dynamic_partition.prefix: 用于指定创建的分区名前缀,例如分区名前缀为p,则自动创建分区名为p20200108 diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java index 51760a4..4d31304 100755 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java @@ -5355,13 +5355,15 @@ public class Catalog { throw new DdlException("not implemented"); } - public void modifyTableDynamicPartition(Database db, OlapTable table, Map<String, String> properties) throws DdlException { + public void modifyTableDynamicPartition(Database db, OlapTable table, Map<String, String> properties) + throws DdlException { Map<String, String> logProperties = new HashMap<>(properties); TableProperty tableProperty = table.getTableProperty(); if (tableProperty == null) { DynamicPartitionUtil.checkAndSetDynamicPartitionProperty(table, properties); } else { - Map<String, String> analyzedDynamicPartition = DynamicPartitionUtil.analyzeDynamicPartition(properties); + Map<String, String> analyzedDynamicPartition = DynamicPartitionUtil. + analyzeDynamicPartition(properties, table.getPartitionInfo()); tableProperty.modifyTableProperties(analyzedDynamicPartition); tableProperty.buildDynamicProperty(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java index 51614cc..bbc2aa8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java @@ -17,6 +17,8 @@ package org.apache.doris.catalog; +import org.apache.doris.common.Config; +import org.apache.doris.common.DdlException; import org.apache.doris.common.FeConstants; import org.apache.doris.common.io.Text; import org.apache.doris.common.io.Writable; @@ -56,7 +58,7 @@ public class TableProperty implements Writable { * DEFAULT: depends on BE's config 'default_rowset_type' * V1: alpha rowset * V2: beta rowset - * + * * This property should be set when creating the table, and can only be changed to V2 using Alter Table stmt. */ private TStorageFormat storageFormat = TStorageFormat.DEFAULT; @@ -77,7 +79,7 @@ public class TableProperty implements Writable { public TableProperty buildProperty(short opCode) { switch (opCode) { case OperationType.OP_DYNAMIC_PARTITION: - buildDynamicProperty(); + executeBuildDynamicProperty(); break; case OperationType.OP_MODIFY_REPLICATION_NUM: buildReplicationNum(); @@ -90,7 +92,19 @@ public class TableProperty implements Writable { } return this; } - public TableProperty buildDynamicProperty() { + public TableProperty buildDynamicProperty() throws DdlException { + if (properties.containsKey(DynamicPartitionProperty.ENABLE) + && Boolean.valueOf(properties.get(DynamicPartitionProperty.ENABLE)) + && !Config.dynamic_partition_enable) { + throw new DdlException("Could not create table with dynamic partition " + + "when fe config dynamic_partition_enable is false. " + + "Please ADMIN SET FRONTEND CONFIG (\"dynamic_partition_enable\" = \"true\") firstly."); + } + executeBuildDynamicProperty(); + return this; + } + + private TableProperty executeBuildDynamicProperty() { HashMap<String, String> dynamicPartitionProperties = new HashMap<>(); for (Map.Entry<String, String> entry : properties.entrySet()) { if (entry.getKey().startsWith(DYNAMIC_PARTITION_PROPERTY_PREFIX)) { @@ -153,7 +167,7 @@ public class TableProperty implements Writable { public static TableProperty read(DataInput in) throws IOException { return GsonUtils.GSON.fromJson(Text.readString(in), TableProperty.class) - .buildDynamicProperty() + .executeBuildDynamicProperty() .buildReplicationNum() .buildInMemory() .buildStorageFormat(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java index 5e68161..56bd051 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java @@ -60,7 +60,7 @@ public class DynamicPartitionUtil { private static final String DATE_FORMAT = "yyyy-MM-dd"; private static final String DATETIME_FORMAT = "yyyy-MM-dd HH:mm:ss"; - public static void checkTimeUnit(String timeUnit) throws DdlException { + public static void checkTimeUnit(String timeUnit, PartitionInfo partitionInfo) throws DdlException { if (Strings.isNullOrEmpty(timeUnit) || !(timeUnit.equalsIgnoreCase(TimeUnit.DAY.toString()) || timeUnit.equalsIgnoreCase(TimeUnit.HOUR.toString()) @@ -68,6 +68,16 @@ public class DynamicPartitionUtil { || timeUnit.equalsIgnoreCase(TimeUnit.MONTH.toString()))) { ErrorReport.reportDdlException(ErrorCode.ERROR_DYNAMIC_PARTITION_TIME_UNIT, timeUnit); } + Preconditions.checkState(partitionInfo instanceof RangePartitionInfo); + RangePartitionInfo rangePartitionInfo = (RangePartitionInfo) partitionInfo; + Preconditions.checkState(!rangePartitionInfo.isMultiColumnPartition()); + Column partitionColumn = rangePartitionInfo.getPartitionColumns().get(0); + if ((partitionColumn.getDataType() == PrimitiveType.DATE) + && (timeUnit.equalsIgnoreCase(TimeUnit.HOUR.toString()))) { + ErrorReport.reportDdlException(DynamicPartitionProperty.TIME_UNIT + " could not be " + + TimeUnit.HOUR.toString() + " when type of partition column " + + partitionColumn.getDisplayName() + " is " + PrimitiveType.DATE.toString()); + } } private static void checkPrefix(String prefix) throws DdlException { @@ -142,8 +152,8 @@ public class DynamicPartitionUtil { throw new DdlException("Invalid properties: " + DynamicPartitionProperty.START_DAY_OF_WEEK); } try { - int dayOfWeek= Integer.parseInt(val); - if (dayOfWeek< 1 || dayOfWeek > 7) { + int dayOfWeek = Integer.parseInt(val); + if (dayOfWeek < 1 || dayOfWeek > 7) { throw new DdlException(DynamicPartitionProperty.START_DAY_OF_WEEK + " should between 1 and 7"); } } catch (NumberFormatException e) { @@ -180,7 +190,7 @@ public class DynamicPartitionUtil { properties.containsKey(DynamicPartitionProperty.START_DAY_OF_MONTH); } - public static boolean checkInputDynamicPartitionProperties(Map<String, String> properties, PartitionInfo partitionInfo) throws DdlException{ + public static boolean checkInputDynamicPartitionProperties(Map<String, String> properties, PartitionInfo partitionInfo) throws DdlException { if (properties == null || properties.isEmpty()) { return false; } @@ -194,13 +204,13 @@ public class DynamicPartitionUtil { String end = properties.get(DynamicPartitionProperty.END); String buckets = properties.get(DynamicPartitionProperty.BUCKETS); String enable = properties.get(DynamicPartitionProperty.ENABLE); - if (!((Strings.isNullOrEmpty(enable) && + if (!(Strings.isNullOrEmpty(enable) && Strings.isNullOrEmpty(timeUnit) && Strings.isNullOrEmpty(timeZone) && Strings.isNullOrEmpty(prefix) && Strings.isNullOrEmpty(start) && Strings.isNullOrEmpty(end) && - Strings.isNullOrEmpty(buckets)))) { + Strings.isNullOrEmpty(buckets))) { if (Strings.isNullOrEmpty(enable)) { properties.put(DynamicPartitionProperty.ENABLE, "true"); } @@ -237,12 +247,12 @@ public class DynamicPartitionUtil { } } - public static Map<String, String> analyzeDynamicPartition(Map<String, String> properties) throws DdlException { + public static Map<String, String> analyzeDynamicPartition(Map<String, String> properties, PartitionInfo partitionInfo) throws DdlException { // properties should not be empty, check properties before call this function Map<String, String> analyzedProperties = new HashMap<>(); if (properties.containsKey(DynamicPartitionProperty.TIME_UNIT)) { String timeUnitValue = properties.get(DynamicPartitionProperty.TIME_UNIT); - checkTimeUnit(timeUnitValue); + checkTimeUnit(timeUnitValue, partitionInfo); properties.remove(DynamicPartitionProperty.TIME_UNIT); analyzedProperties.put(DynamicPartitionProperty.TIME_UNIT, timeUnitValue); } @@ -277,14 +287,14 @@ public class DynamicPartitionUtil { properties.remove(DynamicPartitionProperty.START); analyzedProperties.put(DynamicPartitionProperty.START, startValue); } - + if (properties.containsKey(DynamicPartitionProperty.START_DAY_OF_MONTH)) { String val = properties.get(DynamicPartitionProperty.START_DAY_OF_MONTH); checkStartDayOfMonth(val); properties.remove(DynamicPartitionProperty.START_DAY_OF_MONTH); analyzedProperties.put(DynamicPartitionProperty.START_DAY_OF_MONTH, val); } - + if (properties.containsKey(DynamicPartitionProperty.START_DAY_OF_WEEK)) { String val = properties.get(DynamicPartitionProperty.START_DAY_OF_WEEK); checkStartDayOfWeek(val); @@ -334,9 +344,11 @@ public class DynamicPartitionUtil { /** * properties should be checked before call this method */ - public static void checkAndSetDynamicPartitionProperty(OlapTable olapTable, Map<String, String> properties) throws DdlException { + public static void checkAndSetDynamicPartitionProperty(OlapTable olapTable, Map<String, String> properties) + throws DdlException { if (DynamicPartitionUtil.checkInputDynamicPartitionProperties(properties, olapTable.getPartitionInfo())) { - Map<String, String> dynamicPartitionProperties = DynamicPartitionUtil.analyzeDynamicPartition(properties); + Map<String, String> dynamicPartitionProperties = + DynamicPartitionUtil.analyzeDynamicPartition(properties, olapTable.getPartitionInfo()); TableProperty tableProperty = olapTable.getTableProperty(); if (tableProperty != null) { tableProperty.modifyTableProperties(dynamicPartitionProperties); @@ -391,7 +403,7 @@ public class DynamicPartitionUtil { // add support: HOUR by caoyang10 // TODO: support YEAR public static String getPartitionRangeString(DynamicPartitionProperty property, ZonedDateTime current, - int offset, String format) { + int offset, String format) { String timeUnit = property.getTimeUnit(); if (timeUnit.equalsIgnoreCase(TimeUnit.DAY.toString())) { return getPartitionRangeOfDay(current, offset, format); @@ -408,25 +420,25 @@ public class DynamicPartitionUtil { * return formatted string of partition range in HOUR granularity. * offset: The offset from the current hour. 0 means current hour, 1 means next hour, -1 last hour. * format: the format of the return date string. - * + * <p> * Eg: - * Today is 2020-05-24 00:12:34, offset = -1 - * It will return 2020-05-23 23:00:00 - * Today is 2020-05-24 00, offset = 1 - * It will return 2020-05-24 01:00:00 + * Today is 2020-05-24 00:12:34, offset = -1 + * It will return 2020-05-23 23:00:00 + * Today is 2020-05-24 00, offset = 1 + * It will return 2020-05-24 01:00:00 */ private static String getPartitionRangeOfHour(ZonedDateTime current, int offset, String format) { return getFormattedTimeWithoutMinuteSecond(current.plusHours(offset), format); } - + /** * return formatted string of partition range in DAY granularity. * offset: The offset from the current day. 0 means current day, 1 means tomorrow, -1 means yesterday. * format: the format of the return date string. - * + * <p> * Eg: - * Today is 2020-05-24, offset = -1 - * It will return 2020-05-23 + * Today is 2020-05-24, offset = -1 + * It will return 2020-05-23 */ private static String getPartitionRangeOfDay(ZonedDateTime current, int offset, String format) { return getFormattedTimeWithoutHourMinuteSecond(current.plusDays(offset), format); @@ -437,10 +449,10 @@ public class DynamicPartitionUtil { * offset: The offset from the current week. 0 means current week, 1 means next week, -1 means last week. * startOf: Define the start day of each week. 1 means MONDAY, 7 means SUNDAY. * format: the format of the return date string. - * + * <p> * Eg: - * Today is 2020-05-24, offset = -1, startOf.dayOfWeek = 3 - * It will return 2020-05-20 (Wednesday of last week) + * Today is 2020-05-24, offset = -1, startOf.dayOfWeek = 3 + * It will return 2020-05-20 (Wednesday of last week) */ private static String getPartitionRangeOfWeek(ZonedDateTime current, int offset, StartOfDate startOf, String format) { Preconditions.checkArgument(startOf.isStartOfWeek()); @@ -457,10 +469,10 @@ public class DynamicPartitionUtil { * offset: The offset from the current month. 0 means current month, 1 means next month, -1 means last month. * startOf: Define the start date of each month. 1 means start on the 1st of every month. * format: the format of the return date string. - * + * <p> * Eg: - * Today is 2020-05-24, offset = 1, startOf.month = 3 - * It will return 2020-06-03 + * Today is 2020-05-24, offset = 1, startOf.month = 3 + * It will return 2020-06-03 */ private static String getPartitionRangeOfMonth(ZonedDateTime current, int offset, StartOfDate startOf, String format) { Preconditions.checkArgument(startOf.isStartOfMonth()); diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/util/DynamicPartitionUtilTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/util/DynamicPartitionUtilTest.java index a5e953b..8ad45dd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/common/util/DynamicPartitionUtilTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/common/util/DynamicPartitionUtilTest.java @@ -17,10 +17,16 @@ package org.apache.doris.common.util; +import org.apache.doris.catalog.Column; import org.apache.doris.catalog.DynamicPartitionProperty; +import org.apache.doris.catalog.RangePartitionInfo; +import org.apache.doris.catalog.Type; +import org.apache.doris.common.DdlException; +import org.apache.doris.common.jmockit.Deencapsulation; import com.google.common.collect.Maps; +import com.clearspring.analytics.util.Lists; import org.junit.Assert; import org.junit.Test; @@ -28,6 +34,7 @@ import java.time.DateTimeException; import java.time.LocalDate; import java.time.ZonedDateTime; import java.time.format.DateTimeFormatter; +import java.util.List; import java.util.Map; import java.util.TimeZone; @@ -209,4 +216,21 @@ public class DynamicPartitionUtilTest { Assert.assertEquals("201911", partName); } + @Test + public void testCheckTimeUnit() { + DynamicPartitionUtil dynamicPartitionUtil = new DynamicPartitionUtil(); + RangePartitionInfo rangePartitionInfo = new RangePartitionInfo(); + Deencapsulation.setField(rangePartitionInfo, "isMultiColumnPartition", false); + List<Column> partitionColumnList = Lists.newArrayList(); + Column partitionColumn = new Column(); + partitionColumn.setType(Type.DATE); + Deencapsulation.setField(rangePartitionInfo, partitionColumnList); + try { + Deencapsulation.invoke(dynamicPartitionUtil, "checkTimeUnit", "HOUR", rangePartitionInfo); + Assert.fail(); + } catch (Exception e) { + System.out.print(e.getMessage()); + } + } + } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org