This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 53c4a24cf7a [fix](schema-change) Reject values exceeding the
CHAR/VARCHAR length in bytes (#69061)
53c4a24cf7a is described below
commit 53c4a24cf7a28bfd95b5dd6e33fe5b1a8ace8b21
Author: Jerry Hu <[email protected]>
AuthorDate: Thu Oct 15 14:11:42 2026 +0800
[fix](schema-change) Reject values exceeding the CHAR/VARCHAR length in
bytes (#69061)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
`ALTER TABLE ... MODIFY COLUMN j VARCHAR(n)` on a JSON column runs a
full
schema change that rewrites historical rows with `CAST(j AS
VARCHAR(n))`.
FE cannot bound the text length of JSON, and unlike the load path
(`OlapTableBlockConvertor`), the BE schema change path never validated
the
length of the converted strings. Historical JSON text longer than `n`
was
written into the new VARCHAR(n) column unchanged, while rows inserted
after
the schema change were limited to `n` bytes, so the same column obeyed
different length constraints for old and new rows.
Reproduce:
```sql
CREATE TABLE t (k INT, j JSON) DUPLICATE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t VALUES (1, '{"a": 1}');
ALTER TABLE t MODIFY COLUMN j VARCHAR(1);
-- before: the job FINISHED and LENGTH(j) of the old row was 7
```
`BlockChanger::change_block` now checks, right after the existing
null-map
validation, that every non-null converted value fits the declared
CHAR/VARCHAR length of the new tablet column and fails the alter task
with DATA_QUALITY_ERROR otherwise. The limit comes from the new tablet
column instead of the expression result type, because a string-to-string
change (e.g. widening VARCHAR without light schema change) is sent as a
slot ref whose runtime type carries no length; the 2-byte length prefix
included in the VARCHAR tablet column length is excluded. The job is
cancelled with a message naming the column, the declared length, the
actual length and a value prefix, and the table keeps its original
schema
and data. Users can retry with a sufficient length or STRING. The check
is
shared by local and cloud schema change since both use `BlockChanger`.
Default values of new CHAR/VARCHAR columns had the same gap: FE compared
the default value with the declared length by characters
(`String.length()`), while the length is in bytes. For example
`ADD COLUMN c VARCHAR(1) DEFAULT '中'` finished and every old row read a
3-byte value, and a new CHAR key column rewrote old rows with a value
wider than the CHAR length. FE now checks CHAR/VARCHAR default values by
their UTF-8 byte length (CREATE TABLE and ADD/MODIFY COLUMN), and
`change_block` also checks the materialized default value of a new
column
so a rewrite requested by an FE without the check is cancelled as well.
A schema change that only links the old rowsets (e.g. adding a value
column) never rewrites data, so an over-length default value sent by an
FE without the check during a rolling upgrade is not caught by BE.
The rollup job that builds a synchronous materialized view runs the same
`change_block` and had the same gap. For example
`MAX(CAST(v AS CHAR(1)))` is stored as `substring(CAST(v AS CHAR(1)), 1,
1)`,
which keeps the first character, so '中文' wrote the 3-byte '中' into the
CHAR(1) column, while loading the same value into the materialized view
was rejected. The length check now applies to the expression results of
both schema change and rollup, and such a rollup job is cancelled.
### Release note
Schema changes that convert a column to CHAR/VARCHAR(n) (e.g. JSON to
VARCHAR(n)) are now cancelled when an existing value is longer than n
bytes, instead of silently keeping values that violate the new length.
Creating a synchronous materialized view now fails the same way when an
expression result is longer than its CHAR/VARCHAR column in bytes.
A CHAR/VARCHAR default value longer than the column length in bytes is
now rejected (e.g. `VARCHAR(2) DEFAULT '中'`).
### Check List (For Author)
- Test:
- Unit Test: BlockChangerStringLengthTest, StringDefaultValueLengthTest
- Regression test:
schema_change_p0/test_schema_change_string_length_limit,
plus existing test_type_length_change, test_alter_table_column_nullable,
test_varchar_schema_change, test_dup_schema_value_modify,
test_dup_schema_value_modify4, test_unique_schema_value_modify3,
test_agg_schema_value_modify, all suites of rollup_p0 and mv_p0
- Behavior changed: Yes (such alter jobs are cancelled with an explicit
error message; CHAR/VARCHAR
default values longer than the column in bytes are rejected; such
synchronous materialized
view creation is cancelled)
- Does this need documentation: No
---
be/src/storage/schema_change/schema_change.cpp | 51 +++++++
be/src/storage/schema_change/schema_change.h | 6 +
.../block_changer_string_length_test.cpp | 131 ++++++++++++++++
.../java/org/apache/doris/analysis/ColumnDef.java | 6 +
.../analysis/StringDefaultValueLengthTest.java | 48 ++++++
.../test_schema_change_string_length_limit.out | 63 ++++++++
.../test_schema_change_string_length_limit.groovy | 165 +++++++++++++++++++++
7 files changed, 470 insertions(+)
diff --git a/be/src/storage/schema_change/schema_change.cpp
b/be/src/storage/schema_change/schema_change.cpp
index 6295091eb94..f57aad8ef6f 100644
--- a/be/src/storage/schema_change/schema_change.cpp
+++ b/be/src/storage/schema_change/schema_change.cpp
@@ -44,6 +44,7 @@
#include "core/block/column_with_type_and_name.h"
#include "core/column/column.h"
#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
#include "exec/common/util.hpp"
#include "exec/common/variant_util.h"
#include "exprs/aggregate/aggregate_function.h"
@@ -421,6 +422,11 @@ Status BlockChanger::change_block(Block* ref_block, Block*
new_block) const {
RETURN_IF_ERROR(
_check_cast_valid(ref_column_def.column,
result_tmp_column_def.column));
}
+ // Both the schema change and the rollup (e.g. a synchronous
materialized view on
+ // MAX(CAST(v AS CHAR(1)))) write the expression result into the
new column directly.
+ RETURN_IF_ERROR(check_string_length_within_schema(
+ result_tmp_column_def.column,
*_schema_mapping[idx].new_column,
+ _schema_mapping[idx].new_column->name()));
swap_idx_list.emplace_back(result_tmp_column_idx, idx);
} else if (_schema_mapping[idx].ref_column_idx < 0) {
// new column, write default value
@@ -434,6 +440,11 @@ Status BlockChanger::change_block(Block* ref_block, Block*
new_block) const {
column->insert_duplicate_fields(value, row_num);
}
new_block->get_by_position(idx).column = std::move(column);
+ // An FE before the byte length check of default values may send a
default value
+ // longer than the CHAR/VARCHAR length in bytes.
+ RETURN_IF_ERROR(check_string_length_within_schema(
+ new_block->get_by_position(idx).column,
*_schema_mapping[idx].new_column,
+ _schema_mapping[idx].new_column->name()));
} else {
// same type, just swap column
swap_idx_list.emplace_back(_schema_mapping[idx].ref_column_idx,
idx);
@@ -531,6 +542,46 @@ Status BlockChanger::_check_cast_valid(ColumnPtr
input_column, ColumnPtr output_
return Status::OK();
}
+Status BlockChanger::check_string_length_within_schema(const ColumnPtr& column,
+ const TabletColumn&
new_column,
+ const std::string&
column_name) {
+ // The converted values (e.g. JSON to VARCHAR(n)), the rollup expression
results and the default
+ // values of new columns are not truncated to the declared CHAR/VARCHAR
length, so they must be
+ // checked here like the load path does.
+ // STRING has no declared length.
+ int32_t max_length = 0;
+ if (new_column.type() == FieldType::OLAP_FIELD_TYPE_CHAR) {
+ max_length = new_column.length();
+ } else if (new_column.type() == FieldType::OLAP_FIELD_TYPE_VARCHAR) {
+ // The VARCHAR length of a tablet column includes the 2-byte length
prefix, see
+ // TabletColumn::get_field_length_by_type.
+ max_length = new_column.length() -
cast_set<int32_t>(sizeof(OLAP_VARCHAR_MAX_LENGTH));
+ } else {
+ return Status::OK();
+ }
+ DORIS_CHECK(max_length >= 0) << "invalid length of column " << column_name
<< ": "
+ << new_column.length();
+
+ const auto* nullable = check_and_get_column<ColumnNullable>(column.get());
+ const auto& string_column = assert_cast<const ColumnString&>(
+ nullable == nullptr ? *column : nullable->get_nested_column());
+ const auto* null_map = nullable == nullptr ? nullptr :
nullable->get_null_map_data().data();
+ for (size_t i = 0; i < string_column.size(); ++i) {
+ // NULL has no length, same as the load path.
+ if (null_map != nullptr && null_map[i]) {
+ continue;
+ }
+ const auto value = string_column.get_data_at(i);
+ if (value.size > cast_set<size_t>(max_length)) {
+ return Status::DataQualityError(
+ "the length of value exceeds the length of column {} after
schema change, "
+ "schema length: {}; actual length: {}; first 32 bytes of
value: [{}]",
+ column_name, max_length, value.size, value.to_prefix(32));
+ }
+ }
+ return Status::OK();
+}
+
Status LinkedSchemaChange::process(RowsetReaderSharedPtr rowset_reader,
RowsetWriter* rowset_writer,
BaseTabletSPtr new_tablet, BaseTabletSPtr
base_tablet,
TabletSchemaSPtr base_tablet_schema,
diff --git a/be/src/storage/schema_change/schema_change.h
b/be/src/storage/schema_change/schema_change.h
index 1220e05d818..ac2b0052027 100644
--- a/be/src/storage/schema_change/schema_change.h
+++ b/be/src/storage/schema_change/schema_change.h
@@ -83,6 +83,12 @@ public:
bool has_where() const { return _where_expr != nullptr; }
+ // Returns DataQualityError if any non-null value in `column` is longer
than the declared
+ // length in bytes of `new_column` when it is CHAR/VARCHAR.
+ static Status check_string_length_within_schema(const ColumnPtr& column,
+ const TabletColumn&
new_column,
+ const std::string&
column_name);
+
private:
static Status _check_cast_valid(ColumnPtr ref_column, ColumnPtr
new_column);
diff --git a/be/test/storage/schema_change/block_changer_string_length_test.cpp
b/be/test/storage/schema_change/block_changer_string_length_test.cpp
new file mode 100644
index 00000000000..b78e4e9eab5
--- /dev/null
+++ b/be/test/storage/schema_change/block_changer_string_length_test.cpp
@@ -0,0 +1,131 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include <cstring>
+#include <string>
+
+#include "common/status.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_vector.h"
+#include "storage/olap_define.h"
+#include "storage/schema_change/schema_change.h"
+#include "storage/tablet/tablet_schema.h"
+
+namespace doris {
+
+namespace {
+
+// Builds the new tablet column the same way as the tablet meta of a created
tablet, whose
+// VARCHAR length includes the 2-byte length prefix.
+TabletColumn varchar_column(int32_t declared_length) {
+ return {FieldAggregationMethod::OLAP_FIELD_AGGREGATION_NONE,
FieldType::OLAP_FIELD_TYPE_VARCHAR,
+ true, 1, declared_length + sizeof(OLAP_VARCHAR_MAX_LENGTH)};
+}
+
+TabletColumn char_column(int32_t declared_length) {
+ return {FieldAggregationMethod::OLAP_FIELD_AGGREGATION_NONE,
FieldType::OLAP_FIELD_TYPE_CHAR,
+ true, 1, static_cast<size_t>(declared_length)};
+}
+
+} // namespace
+
+TEST(BlockChangerStringLengthTest, AcceptsValuesWithinSchemaLength) {
+ auto column = ColumnString::create();
+ column->insert_data("abc", 3);
+ column->insert_data("abcdef", 6);
+ column->insert_data("", 0);
+
EXPECT_TRUE(BlockChanger::check_string_length_within_schema(std::move(column),
+
varchar_column(6), "c")
+ .ok());
+}
+
+TEST(BlockChangerStringLengthTest, RejectsValueLongerThanSchemaLength) {
+ auto column = ColumnString::create();
+ column->insert_data("abc", 3);
+ column->insert_data(R"({"a":1})", 7);
+ Status st =
BlockChanger::check_string_length_within_schema(std::move(column),
+
varchar_column(5), "j");
+ EXPECT_EQ(st.code(), ErrorCode::DATA_QUALITY_ERROR);
+ EXPECT_NE(st.to_string().find("exceeds the length of column j"),
std::string::npos);
+ EXPECT_NE(st.to_string().find("schema length: 5"), std::string::npos);
+ EXPECT_NE(st.to_string().find("actual length: 7"), std::string::npos);
+ EXPECT_NE(st.to_string().find(R"([{"a":1}])"), std::string::npos);
+}
+
+TEST(BlockChangerStringLengthTest, ExcludesVarcharLengthPrefix) {
+ // VARCHAR(5) has tablet column length 7; values of 6 and 7 bytes must
still be rejected.
+ for (const auto* value : {"abcdef", "abcdefg"}) {
+ auto column = ColumnString::create();
+ column->insert_data(value, strlen(value));
+
EXPECT_EQ(BlockChanger::check_string_length_within_schema(std::move(column),
+
varchar_column(5), "c")
+ .code(),
+ ErrorCode::DATA_QUALITY_ERROR)
+ << value;
+ }
+}
+
+TEST(BlockChangerStringLengthTest, ChecksCharLength) {
+ auto column = ColumnString::create();
+ column->insert_data("abcde", 5);
+ ColumnPtr ptr = std::move(column);
+ EXPECT_TRUE(BlockChanger::check_string_length_within_schema(ptr,
char_column(5), "c").ok());
+ EXPECT_EQ(BlockChanger::check_string_length_within_schema(ptr,
char_column(4), "c").code(),
+ ErrorCode::DATA_QUALITY_ERROR);
+}
+
+TEST(BlockChangerStringLengthTest, IgnoresStringColumn) {
+ // STRING has no declared length.
+ auto column = ColumnString::create();
+ column->insert_data("abcdefgh", 8);
+ TabletColumn
string_column(FieldAggregationMethod::OLAP_FIELD_AGGREGATION_NONE,
+ FieldType::OLAP_FIELD_TYPE_STRING, true, 1, 1);
+ EXPECT_TRUE(
+ BlockChanger::check_string_length_within_schema(std::move(column),
string_column, "c")
+ .ok());
+}
+
+TEST(BlockChangerStringLengthTest, ComparesByteLength) {
+ // "中文" is 6 bytes in UTF-8, VARCHAR length is in bytes.
+ auto column = ColumnString::create();
+ column->insert_data("中文", 6);
+ ColumnPtr ptr = std::move(column);
+ EXPECT_TRUE(BlockChanger::check_string_length_within_schema(ptr,
varchar_column(6), "c").ok());
+ EXPECT_EQ(BlockChanger::check_string_length_within_schema(ptr,
varchar_column(5), "c").code(),
+ ErrorCode::DATA_QUALITY_ERROR);
+}
+
+TEST(BlockChangerStringLengthTest, SkipsNullRowsButChecksNonNullRows) {
+ // The nested payload of a NULL row is ignored even if it is long.
+ auto nested = ColumnString::create();
+ nested->insert_data("zzzzzzzzzz", 10);
+ nested->insert_data("ab", 2);
+ auto null_map = ColumnUInt8::create();
+ null_map->insert_value(1);
+ null_map->insert_value(0);
+ ColumnPtr nullable = ColumnNullable::create(std::move(nested),
std::move(null_map));
+ EXPECT_TRUE(
+ BlockChanger::check_string_length_within_schema(nullable,
varchar_column(2), "c").ok());
+ EXPECT_EQ(BlockChanger::check_string_length_within_schema(nullable,
varchar_column(1), "c")
+ .code(),
+ ErrorCode::DATA_QUALITY_ERROR);
+}
+
+} // namespace doris
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
index fa2b593e5d4..df8046dc33c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
@@ -34,6 +34,7 @@ import org.apache.doris.nereids.types.DataType;
import com.google.common.base.Preconditions;
+import java.nio.charset.StandardCharsets;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.HashSet;
@@ -440,6 +441,11 @@ public class ColumnDef {
break;
case CHAR:
case VARCHAR:
+ // CHAR/VARCHAR length is in bytes, the same as the load path
checks.
+ if (defaultValue.getBytes(StandardCharsets.UTF_8).length >
scalarType.getLength()) {
+ throw new AnalysisException("Default value is too long: "
+ defaultValue);
+ }
+ break;
case HLL:
case STRING:
case JSONB:
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/analysis/StringDefaultValueLengthTest.java
b/fe/fe-core/src/test/java/org/apache/doris/analysis/StringDefaultValueLengthTest.java
new file mode 100644
index 00000000000..a4a99c32ff0
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/analysis/StringDefaultValueLengthTest.java
@@ -0,0 +1,48 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.AnalysisException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+class StringDefaultValueLengthTest {
+ // CHAR/VARCHAR length is in bytes, "中" is 3 bytes in UTF-8.
+ @Test
+ void testVarcharDefaultValueLengthInBytes() {
+ Assertions.assertDoesNotThrow(
+ () ->
ColumnDef.validateDefaultValue(ScalarType.createVarcharType(3), "中", null));
+ AnalysisException e = Assertions.assertThrows(AnalysisException.class,
+ () ->
ColumnDef.validateDefaultValue(ScalarType.createVarcharType(2), "中", null));
+ Assertions.assertTrue(e.getMessage().contains("Default value is too
long"), e.getMessage());
+ }
+
+ @Test
+ void testCharDefaultValueLengthInBytes() {
+ Assertions.assertDoesNotThrow(
+ () ->
ColumnDef.validateDefaultValue(ScalarType.createCharType(3), "中", null));
+ Assertions.assertDoesNotThrow(
+ () ->
ColumnDef.validateDefaultValue(ScalarType.createCharType(3), "abc", null));
+ Assertions.assertThrows(AnalysisException.class,
+ () ->
ColumnDef.validateDefaultValue(ScalarType.createCharType(1), "中", null));
+ Assertions.assertThrows(AnalysisException.class,
+ () ->
ColumnDef.validateDefaultValue(ScalarType.createCharType(3), "abcd", null));
+ }
+}
diff --git
a/regression-test/data/schema_change_p0/test_schema_change_string_length_limit.out
b/regression-test/data/schema_change_p0/test_schema_change_string_length_limit.out
new file mode 100644
index 00000000000..97f64162cd0
--- /dev/null
+++
b/regression-test/data/schema_change_p0/test_schema_change_string_length_limit.out
@@ -0,0 +1,63 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !before --
+1 {"a":1}
+2 [1,2,3]
+3 \N
+
+-- !desc_after_cancel --
+k int Yes true \N
+j json Yes false \N NONE
+
+-- !after_cancel --
+1 {"a":1}
+2 [1,2,3]
+3 \N
+
+-- !desc_after_finish --
+k int Yes true \N
+j varchar(7) Yes false \N NONE
+
+-- !after_finish --
+1 {"a":1} 7
+2 [1,2,3] 7
+3 \N \N
+4 abcdefg 7
+
+-- !desc_after_widen --
+k int Yes true \N
+v varchar(6) Yes false \N NONE
+
+-- !after_widen --
+1 abcde 5
+2 abc 3
+3 \N \N
+4 abcdef 6
+
+-- !desc_after_default --
+k int Yes true \N
+ck char(3) Yes true 中
+v int Yes false \N NONE
+c varchar(3) Yes false 中 NONE
+
+-- !after_default --
+1 中 3 1 中 3
+2 中 3 2 中 3
+
+-- !mv_char --
+1 x 1
+2 d 1
+3 \N \N
+
+-- !mv_varchar --
+1 {" 2
+2 \N \N
+3 \N \N
+
+-- !desc_after_mv_cancel --
+test_schema_change_string_length_limit_mv DUP_KEYS k int
int Yes true \N true
+ v varchar(8192) varchar(8192) Yes false \N
NONE true
+ j json json Yes false \N NONE true
+
+test_schema_change_string_length_limit_mv_varchar AGG_KEYS kv
int int Yes true \N true `k`
+ v2 varchar(2) varchar(2) Yes false \N
MAX true substring(CAST(`j` AS varchar(2)), 1, 2)
+
diff --git
a/regression-test/suites/schema_change_p0/test_schema_change_string_length_limit.groovy
b/regression-test/suites/schema_change_p0/test_schema_change_string_length_limit.groovy
new file mode 100644
index 00000000000..ea75ad8e295
--- /dev/null
+++
b/regression-test/suites/schema_change_p0/test_schema_change_string_length_limit.groovy
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_schema_change_string_length_limit", "p0") {
+ def waitForJob = { tableName, expectedState ->
+ def row = null
+ int maxTrySecs = 120
+ while (maxTrySecs-- > 0) {
+ def res = sql """ SHOW ALTER TABLE COLUMN WHERE
IndexName='${tableName}' ORDER BY createtime DESC LIMIT 1 """
+ row = res[0]
+ if (row[9] == "FINISHED" || row[9] == "CANCELLED") {
+ break
+ }
+ sleep(1000)
+ }
+ assertEquals(expectedState, row[9])
+ return row
+ }
+
+ sql "DROP TABLE IF EXISTS test_schema_change_string_length_limit"
+ sql """
+ CREATE TABLE test_schema_change_string_length_limit (
+ k INT,
+ j JSON
+ ) DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ // Both JSON values are 7 bytes after being converted to text, the NULL
row has no length.
+ sql """INSERT INTO test_schema_change_string_length_limit VALUES (1,
'{"a": 1}'), (2, '[1,2,3]'), (3, NULL)"""
+ order_qt_before """SELECT k, j FROM
test_schema_change_string_length_limit"""
+
+ // The new length is too short for the old data: the job is cancelled and
the table is unchanged.
+ sql """ALTER TABLE test_schema_change_string_length_limit MODIFY COLUMN j
VARCHAR(5)"""
+ def cancelled = waitForJob("test_schema_change_string_length_limit",
"CANCELLED")
+ assertTrue(cancelled[10].toString().contains("exceeds the length of column
j"), cancelled[10].toString())
+ qt_desc_after_cancel """DESC test_schema_change_string_length_limit"""
+ order_qt_after_cancel """SELECT k, j FROM
test_schema_change_string_length_limit"""
+
+ // The new length is enough: the job finishes and old and new rows obey
VARCHAR(7).
+ sql """ALTER TABLE test_schema_change_string_length_limit MODIFY COLUMN j
VARCHAR(7)"""
+ waitForJob("test_schema_change_string_length_limit", "FINISHED")
+ qt_desc_after_finish """DESC test_schema_change_string_length_limit"""
+ sql """INSERT INTO test_schema_change_string_length_limit VALUES (4,
'abcdefg')"""
+ order_qt_after_finish """SELECT k, j, LENGTH(j) FROM
test_schema_change_string_length_limit"""
+
+ // Widening VARCHAR on a table without light schema change rewrites old
data through a slot
+ // ref, whose values already fit the new length.
+ sql "DROP TABLE IF EXISTS test_schema_change_string_length_limit_widen"
+ sql """
+ CREATE TABLE test_schema_change_string_length_limit_widen (
+ k INT,
+ v VARCHAR(5)
+ ) DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES ("replication_num" = "1", "light_schema_change" = "false")
+ """
+ sql """INSERT INTO test_schema_change_string_length_limit_widen VALUES (1,
'abcde'), (2, 'abc'), (3, NULL)"""
+ sql """ALTER TABLE test_schema_change_string_length_limit_widen MODIFY
COLUMN v VARCHAR(6)"""
+ waitForJob("test_schema_change_string_length_limit_widen", "FINISHED")
+ qt_desc_after_widen """DESC test_schema_change_string_length_limit_widen"""
+ sql """INSERT INTO test_schema_change_string_length_limit_widen VALUES (4,
'abcdef')"""
+ order_qt_after_widen """SELECT k, v, LENGTH(v) FROM
test_schema_change_string_length_limit_widen"""
+
+ // CHAR/VARCHAR length is in bytes, so a default value longer than the
column in bytes is
+ // rejected. '中' is 3 bytes in UTF-8.
+ sql "DROP TABLE IF EXISTS test_schema_change_string_length_limit_default"
+ test {
+ sql """
+ CREATE TABLE test_schema_change_string_length_limit_default (
+ k INT,
+ c VARCHAR(2) DEFAULT '中'
+ ) DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ exception "Default value is too long"
+ }
+ sql """
+ CREATE TABLE test_schema_change_string_length_limit_default (
+ k INT,
+ v INT
+ ) DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES ("replication_num" = "1", "light_schema_change" = "false")
+ """
+ sql """INSERT INTO test_schema_change_string_length_limit_default VALUES
(1, 1), (2, 2)"""
+ // A new value column is linked and filled with the default value when it
is read.
+ test {
+ sql """ALTER TABLE test_schema_change_string_length_limit_default ADD
COLUMN c VARCHAR(2) DEFAULT '中'"""
+ exception "Default value is too long"
+ }
+ // A new CHAR key column changes the short key and rewrites old data with
the default value.
+ test {
+ sql """ALTER TABLE test_schema_change_string_length_limit_default ADD
COLUMN ck CHAR(1) KEY DEFAULT '中' AFTER k"""
+ exception "Default value is too long"
+ }
+ sql """ALTER TABLE test_schema_change_string_length_limit_default ADD
COLUMN c VARCHAR(3) DEFAULT '中'"""
+ waitForJob("test_schema_change_string_length_limit_default", "FINISHED")
+ sql """ALTER TABLE test_schema_change_string_length_limit_default ADD
COLUMN ck CHAR(3) KEY DEFAULT '中' AFTER k"""
+ waitForJob("test_schema_change_string_length_limit_default", "FINISHED")
+ qt_desc_after_default """DESC
test_schema_change_string_length_limit_default"""
+ order_qt_after_default """SELECT k, ck, LENGTH(ck), v, c, LENGTH(c) FROM
test_schema_change_string_length_limit_default"""
+
+ // A synchronous materialized view is built by the rollup job, which also
checks the length of
+ // the expression results.
+ sql "DROP TABLE IF EXISTS test_schema_change_string_length_limit_mv"
+ sql """
+ CREATE TABLE test_schema_change_string_length_limit_mv (
+ k INT,
+ v VARCHAR(8192),
+ j JSON
+ ) DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """INSERT INTO test_schema_change_string_length_limit_mv VALUES
+ (1, repeat('x', 8192), '{"a": 12345}'), (1, 'abc', '[1,2,3]'), (2,
'de', NULL), (3, NULL, NULL)"""
+ // Values cast to a narrow CHAR/VARCHAR fit the materialized view column.
+ create_sync_mv(context.dbName, "test_schema_change_string_length_limit_mv",
+ "test_schema_change_string_length_limit_mv_char",
+ """SELECT k AS kc, MAX(CAST(v AS CHAR(1))) AS c1 FROM
test_schema_change_string_length_limit_mv GROUP BY k""")
+ create_sync_mv(context.dbName, "test_schema_change_string_length_limit_mv",
+ "test_schema_change_string_length_limit_mv_varchar",
+ """SELECT k AS kv, MAX(CAST(j AS VARCHAR(2))) AS v2 FROM
test_schema_change_string_length_limit_mv GROUP BY k""")
+ order_qt_mv_char """SELECT kc, c1, LENGTH(c1) FROM
test_schema_change_string_length_limit_mv INDEX
test_schema_change_string_length_limit_mv_char"""
+ order_qt_mv_varchar """SELECT kv, v2, LENGTH(v2) FROM
test_schema_change_string_length_limit_mv INDEX
test_schema_change_string_length_limit_mv_varchar"""
+
+ // CAST(v AS CHAR(1)) keeps the first character, which is 3 bytes for '中文'
and longer than the
+ // CHAR(1) column in bytes, so the rollup job is cancelled like a load of
the same value into the
+ // materialized view fails.
+ sql """DROP MATERIALIZED VIEW IF EXISTS
test_schema_change_string_length_limit_mv_char ON
test_schema_change_string_length_limit_mv"""
+ sql """INSERT INTO test_schema_change_string_length_limit_mv VALUES (4,
'中文', NULL)"""
+ sql """CREATE MATERIALIZED VIEW
test_schema_change_string_length_limit_mv_char AS
+ SELECT k AS kc, MAX(CAST(v AS CHAR(1))) AS c1 FROM
test_schema_change_string_length_limit_mv GROUP BY k"""
+ def mvRow = null
+ for (int i = 0; i < 120; i++) {
+ // The latest job of the materialized view, an earlier job of the same
name has finished.
+ mvRow = sql(""" SHOW ALTER TABLE MATERIALIZED VIEW WHERE
TableName='test_schema_change_string_length_limit_mv' """)
+ .findAll { it[5] ==
"test_schema_change_string_length_limit_mv_char" }
+ .max { it[0] as Long }
+ if (mvRow[8] == "FINISHED" || mvRow[8] == "CANCELLED") {
+ break
+ }
+ sleep(1000)
+ }
+ assertEquals("CANCELLED", mvRow[8])
+ assertTrue(mvRow[9].toString().contains("exceeds the length of column"),
mvRow[9].toString())
+ // Only the VARCHAR materialized view is left.
+ qt_desc_after_mv_cancel """DESC test_schema_change_string_length_limit_mv
ALL"""
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
