Doris-Breakwater commented on issue #67349: URL: https://github.com/apache/doris/issues/67349#issuecomment-5477620664
Breakwater-GitHub-Analysis-Slot: slot_9a8784d19205 ## Initial assessment Confirmed FE/MySQL-protocol metadata bug with deterministic client impact. This is not a server-side row-data truncation issue: the incorrect value is produced while FE serializes `Protocol::ColumnDefinition41`, before the correctly returned row bytes are consumed by the client. The issue currently has no labels; `kind/fix`, `area/mysql-compatibility`, `area/odbc`, and `kind/need-regression-test` are appropriate. ## Verified facts * On the current public `branch-3.1` tip, all three `MysqlSerializer.writeField(...)` overloads write the four-byte `column_length` through `getMysqlTypeLength(...)` ([code](https://github.com/apache/doris/blob/a3e9f66a1106a4208e6855ffef9e0fb8b3c981ff/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlSerializer.java#L164-L255)). `CHAR` and `VARCHAR` have no cases in that helper and therefore reach `default: return 255` ([code](https://github.com/apache/doris/blob/a3e9f66a1106a4208e6855ffef9e0fb8b3c981ff/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlSerializer.java#L265-L319)). The same helper remains in the referenced master commit. * The declared length has not been lost upstream. Normal result-set metadata passes each result expression's full `Type` into the serializer ([code](https://github.com/apache/doris/blob/a3e9f66a1106a4208e6855ffef9e0fb8b3c981ff/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L2878-L2918), [type extraction](https://github.com/apache/doris/blob/a3e9f66a1106a4208e6855ffef9e0fb8b3c981ff/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L3494-L3496)); the prepared-statement path also passes either the original `Column` or the output slot type ([code](https://github.com/apache/doris/blob/a3e9f66a1106a4208e6855ffef9e0fb8b3c981ff/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L2847-L2861)). `ScalarType` stores the declared `VARCHAR` length and exposes it through `getLength()` ([code](https://github.com/apache/doris/blob/a3e9f66a1106a4208e6855ffef9e0fb8b3c981ff/fe/fe-common/src/main/java/org/apache/doris/catalog/ScalarType.java#L524-L528), [getter ](https://github.com/apache/doris/blob/a3e9f66a1106a4208e6855ffef9e0fb8b3c981ff/fe/fe-common/src/main/java/org/apache/doris/catalog/ScalarType.java#L786-L798)). Therefore the constant is the direct cause; no catalog or BE change is needed for the reported direct-column case. * This is a longstanding compatibility limitation rather than evidence of a recent 3.1 regression: the string fallback has existed since the type-aware helper was introduced, and the prior implementation wrote `255` directly. * Current master even has a serializer test that explicitly expects `255` for `VARCHAR(10)` ([test](https://github.com/apache/doris/blob/ded08aebefdb76b167c1f5fa164feaa1b4732205/fe/fe-core/src/test/java/org/apache/doris/mysql/MysqlSerializerVarbinaryTest.java#L101-L120)); that expectation must change with a fix. * The other packet differences are real but independent. `branch-3.1` writes collation 33 unconditionally in these overloads, and `VARCHAR` falls through to `MYSQL_TYPE_STRING` in `PrimitiveType.toMysqlType()` ([code](https://github.com/apache/doris/blob/a3e9f66a1106a4208e6855ffef9e0fb8b3c981ff/fe/fe-common/src/main/java/org/apache/doris/catalog/PrimitiveType.java#L958-L1008)). Neither difference causes the hardcoded `255`. ## Important byte/character-length caveat The minimal code defect is clear, but `return type.getLength()` should not be merged without an ODBC assertion. MySQL defines `column_length` as the field's maximum length, while ODBC `SQLDescribeCol` exposes character count. Connector/ODBC 8.1 derives that count for `VARCHAR`/`VAR_STRING`/`STRING` by dividing the packet length by the advertised charset's `mbmaxlen` ([driver source](https://github.com/mysql/mysql-connector-odbc/blob/8.1.0/driver/utility.cc#L1477-L1554)). Doris 3.1 advertises collation 33 (`utf8mb3`) even though Doris `VARCHAR(M)` is byte-limited. Consequently, returning the Doris byte limit `M` fixes the literal packet value requested in this issue, but a multibyte-aware driver can still expose a `ColumnSize` smaller than `M`; that may leave the reported silent-truncation scenario unresolved. Maintainers should first define the compatibility contract: either report the actual Doris maximum bytes, or conservatively encode a protocol length that makes character-oriented clients allocate for up to `M` single-byte characters. The chosen formula must be tested with Connector/ODBC rather than inferred from `mysql --column-type-info` alone. Changing the type code or collation is a broader compatibility change and should be evaluated separately, not bundled without a driver matrix. ## Missing information No FE log or query profile is needed to confirm the core bug. Two items are useful before finalizing the client-facing behavior: 1. The build suffix `9378ac80` does not resolve to a commit in the public `apache/doris` repository. A full public SHA or build provenance would allow exact version/backport verification, although public `branch-3.1` contains the same faulty logic. 2. To explain the exact reported `ColumnSize = 127` and select the safe length formula, please provide the redacted DSN character-set/options plus values obtained directly from `SQLDescribeColW`/`SQLColAttribute` for `SQL_DESC_LENGTH` and `SQL_DESC_OCTET_LENGTH`. `.NET GetSchemaTable()` can add another metadata-conversion layer. ## Recommended next steps 1. Fix `getMysqlTypeLength` with explicit `CHAR` and `VARCHAR` cases; do not replace the broad default with `type.getLength()`, because unrelated fallback types also use it. 2. Add packet-level FE tests for `VARCHAR(10)`, `VARCHAR(1000)`, `VARCHAR(65533)`, `CHAR`, and unspecified-length expression types, and update the existing master test that locks in `255`. Cover the `Type`, `FieldInfo + Type`, and `Column` serializer overloads. 3. Add an end-to-end metadata regression for direct columns and `CAST(... AS VARCHAR(n))`, including text protocol and prepared statements. Validate both the packet length and Connector/ODBC 8.1 Unicode `SQLDescribeCol`, then fetch a value longer than 255 bytes into a metadata-sized buffer to prove there is no truncation. 4. Backport after the packet/ODBC contract is settled. Track collation negotiation and `MYSQL_TYPE_STRING` versus `MYSQL_TYPE_VAR_STRING` as separate compatibility work unless the end-to-end test proves either is required for this fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
