mrhhsg opened a new pull request, #69061:
URL: https://github.com/apache/doris/pull/69061
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
`ALTER TABLE ... MODIFY COLUMN j VARCHAR(n)` on a JSON column runs a full
schema change that rewrites historical rows with `CAST(j AS VARCHAR(n))`.
FE cannot bound the text length of JSON, and unlike the load path
(`OlapTableBlockConvertor`), the BE schema change path never validated the
length of the converted strings. Historical JSON text longer than `n` was
written into the new VARCHAR(n) column unchanged, while rows inserted after
the schema change were limited to `n` bytes, so the same column obeyed
different length constraints for old and new rows.
Reproduce:
```sql
CREATE TABLE t (k INT, j JSON) DUPLICATE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t VALUES (1, '{"a": 1}');
ALTER TABLE t MODIFY COLUMN j VARCHAR(1);
-- before: the job FINISHED and LENGTH(j) of the old row was 7
```
`BlockChanger::change_block` now checks, right after the existing null-map
validation, that every non-null converted value fits the declared
CHAR/VARCHAR length of the new tablet column and fails the alter task
with DATA_QUALITY_ERROR otherwise. The limit comes from the new tablet
column instead of the expression result type, because a string-to-string
change (e.g. widening VARCHAR without light schema change) is sent as a
slot ref whose runtime type carries no length; the 2-byte length prefix
included in the VARCHAR tablet column length is excluded. The job is
cancelled with a message naming the column, the declared length, the
actual length and a value prefix, and the table keeps its original schema
and data. Users can retry with a sufficient length or STRING. The check is
shared by local and cloud schema change since both use `BlockChanger`.
Default values of new CHAR/VARCHAR columns had the same gap: FE compared
the default value with the declared length by characters
(`String.length()`), while the length is in bytes. For example
`ADD COLUMN c VARCHAR(1) DEFAULT '中'` finished and every old row read a
3-byte value, and a new CHAR key column rewrote old rows with a value
wider than the CHAR length. FE now checks CHAR/VARCHAR default values by
their UTF-8 byte length (CREATE TABLE and ADD/MODIFY COLUMN), and
`change_block` also checks the materialized default value of a new column
so a rewrite requested by an FE without the check is cancelled as well.
A schema change that only links the old rowsets (e.g. adding a value
column) never rewrites data, so an over-length default value sent by an
FE without the check during a rolling upgrade is not caught by BE.
The rollup job that builds a synchronous materialized view runs the same
`change_block` and had the same gap. For example
`MAX(CAST(v AS CHAR(1)))` is stored as `substring(CAST(v AS CHAR(1)), 1, 1)`,
which keeps the first character, so '中文' wrote the 3-byte '中' into the
CHAR(1) column, while loading the same value into the materialized view
was rejected. The length check now applies to the expression results of
both schema change and rollup, and such a rollup job is cancelled.
### Release note
Schema changes that convert a column to CHAR/VARCHAR(n) (e.g. JSON to
VARCHAR(n)) are now cancelled when an existing value is longer than n
bytes, instead of silently keeping values that violate the new length.
Creating a synchronous materialized view now fails the same way when an
expression result is longer than its CHAR/VARCHAR column in bytes.
A CHAR/VARCHAR default value longer than the column length in bytes is
now rejected (e.g. `VARCHAR(2) DEFAULT '中'`).
### Check List (For Author)
- Test:
- Unit Test: BlockChangerStringLengthTest, StringDefaultValueLengthTest
- Regression test:
schema_change_p0/test_schema_change_string_length_limit,
plus existing test_type_length_change,
test_alter_table_column_nullable,
test_varchar_schema_change, test_dup_schema_value_modify,
test_dup_schema_value_modify4, test_unique_schema_value_modify3,
test_agg_schema_value_modify, all suites of rollup_p0 and mv_p0
- Behavior changed: Yes (such alter jobs are cancelled with an explicit
error message; CHAR/VARCHAR
default values longer than the column in bytes are rejected; such
synchronous materialized
view creation is cancelled)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
