morningman opened a new pull request, #69045:
URL: https://github.com/apache/doris/pull/69045
### What problem does this PR solve?
Issue Number: None
Related PR: #66227, #68825
Problem Summary:
**In short.** When Doris creates or alters a Paimon table, master drops the
declared length of CHAR(n) and VARCHAR(n) columns (they all become
VARCHAR(2147483647)) and gives DATETIME a precision of 6. Paimon relies on
these bounds when it reads files written under an older schema, so a table
created or altered through Doris does not behave like the schema the user
declared. This PR keeps the bounds, as branch-4.1 #66227 does.
**Background.** `PaimonTypeMapping.toPaimonType` (fe-connector-paimon) turns
the column types of `CREATE TABLE`, `ADD COLUMN` and `MODIFY COLUMN` into the
Paimon schema. Paimon stores that schema with every data file. When it reads a
file written under an older schema, it casts each column from the file's type
to the current type, so the current type decides what a reader sees: a
VARCHAR(3) column truncates older longer values, and a TIMESTAMP(3) column
reads a numeric string such as `42` as milliseconds. The mapping was ported
from fe-core's `DorisToPaimonTypeVisitor` when Paimon moved into the connector
plugin; branch-4.1 #66227 later changed that visitor to keep the bounds, and
the change never reached master. #68825 (the master port of the rest of #66227)
updated this method's Javadoc to describe the new mapping, but not the code.
**The problem, and what it cost.**
| Doris column type | Paimon type on master | Paimon type with this PR |
What changes for users |
|---|---|---|---|
| CHAR(n) | VARCHAR(2147483647) | CHAR(n) | The column reads back as CHAR(n)
instead of STRING. |
| VARCHAR(n) | VARCHAR(2147483647) | VARCHAR(n) | The column reads back as
VARCHAR(n) instead of STRING. `MODIFY COLUMN c VARCHAR(3)` narrows the column;
on master it widens it to the maximum length, and older values are never
truncated. |
| STRING | VARCHAR(2147483647) | unchanged | |
| DATETIME (v1, only when date conversion is disabled) | TIMESTAMP(6) |
TIMESTAMP(0) | |
| DATETIMEV2(p) | TIMESTAMP(p) | unchanged | |
The same applies inside ARRAY, MAP and STRUCT, and to `CREATE TABLE ... AS
SELECT`: query columns that Doris' CTAS rules keep as VARCHAR(n) (columns of
internal tables, and expressions as VARCHAR(65533) by default) keep that length
instead of becoming STRING. Only Doris DDL uses this mapping: `ADD COLUMN` and
`MODIFY COLUMN` on a table created by Spark are affected, reading a table is
not.
**How this PR fixes it.** `toPaimonType` maps CHAR(n) to `CharType(n)`,
VARCHAR(n) to `VarCharType(n)` (CREATE TABLE already carries the length in
`ConnectorType.precision`), STRING to `VarCharType(MAX_LENGTH)`, DATETIME to
`TimestampType(0)` and DATETIMEV2(p) to `TimestampType(p)`. A column declared
as plain `varchar` or `char` is normalized to VARCHAR(65533) or CHAR(1) by
`ColumnDefinition` before it reaches the connector, so the length is always
set. The Javadoc is the same text as in #68825.
**Results.** `PaimonTypeMappingToPaimonTest` now checks the bounds and
precisions, the nested types, and Paimon's own casts on the mapped types: from
VARCHAR(10) to the mapped VARCHAR(3), `abcdef` becomes `abc`; from STRING to
the mapped DATETIMEV2(3), `42` becomes `1970-01-01T00:00:00.042`. With master's
mapping all four of these cases fail. All fe-connector-paimon unit tests (726)
pass. No existing regression suite creates Paimon tables with CHAR(n),
VARCHAR(n) or DATETIME v1 columns through Doris; the
`test_paimon_rust_reader_eq_for_null` suite in #68825 narrows a column to
VARCHAR(3) and needs this change.
Original author: @pzhdfy (#66227)
### Release note
Paimon tables created or altered by Doris keep the declared length of
CHAR(n) and VARCHAR(n) columns instead of turning them into STRING.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `PaimonTypeMappingToPaimonTest`: 4 cases fail with master's
mapping and pass with this PR; all fe-connector-paimon tests (726) pass.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- CHAR(n) / VARCHAR(n) columns of Paimon tables created or altered
by Doris keep their length; DATETIME v1 maps to TIMESTAMP(0).
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
