Mike0808 opened a new issue, #67349: URL: https://github.com/apache/doris/issues/67349
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Doris version doris-3.1.1-rc01-9378ac80 Client side: MySQL Connector/ODBC 8.1 Unicode Driver (Windows) mysql CLI (Linux) ### What's Wrong? When a result set is returned over the MySQL wire protocol, the FE puts a constant 255 into the length field of every column definition packet for VARCHAR columns, regardless of the length declared in the DDL. The actual row data is transmitted correctly and in full — only the metadata is wrong. The mismatch is visible directly in the --column-type-info output, where Length stays at 255 while Max_length correctly reports 900: Field 3: `c_1000` Type: STRING Collation: utf8mb3_general_ci (33) Length: 255 <-- declared VARCHAR(1000) Max_length: 900 Field 7: `c_65533` Type: STRING Collation: utf8mb3_general_ci (33) Length: 255 <-- declared VARCHAR(65533) Max_length: 900 There is also an internal inconsistency inside a single ODBC session: SQLColumns (catalog metadata) returns the correct declared lengths, while SQLDescribeCol (result set metadata) returns a value derived from the hardcoded 255. Practical impact Client applications that size their buffers from result set metadata silently truncate string values. With a Unicode ODBC driver the reported 255 bytes are divided by the wide-character size, producing ColumnSize = 127, so values are cut at roughly 255–256 bytes: Doris reports length = 255 -> Connector/ODBC 8.1(w) reports ColumnSize = 127 -> client allocates a 127-character buffer -> string values are truncated This was originally observed as silent truncation of long text fields in a BI tool loading data through the MySQL ODBC driver. No error or warning is raised on either side — the data simply arrives incomplete. There is no server-side workaround. CAST(col AS VARCHAR(4000)) does not change the reported length, and narrowing the column with ALTER TABLE ... MODIFY COLUMN has no effect either, since 255 is reported unconditionally. ### What You Expected? The length field of the column definition packet should reflect the declared column length (in bytes, consistent with Doris VARCHAR(M) semantics), the same way MySQL does. SQLDescribeCol and SQLColumns should agree with each other. ### How to Reproduce? <html> <body> <!--StartFragment--><h3 dir="ltr">1. Create the table and insert data</h3> <div role="group" aria-label="sql code" tabindex="0"><div><div></div></div><div>sql</div><div><pre style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono);"><code class="language-sql" style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono); white-space: pre;"><span><span><span style="color: rgb(129, 0, 194);">CREATE</span> <span style="color: rgb(129, 0, 194);">DATABASE</span> <span style="color: rgb(0, 81, 194);">IF</span> <span style="color: rgb(129, 0, 194);">NOT</span> <span style="color: rgb(129, 0, 194);">EXISTS</span> odbc_test; </span><span><span style="color: rgb(129, 0, 194);">USE</span> odbc_test; </span><span> </span><span><span style="color: rgb(129, 0, 194);">CREATE</span> <span style="color: rgb(129, 0, 194);">TABLE</span> <span style="color: rgb(0, 81, 194);">t_types</span> ( </span><span> id <span style="color: rgb(129, 0, 194);">INT</span> <span style="color: rgb(129, 0, 194);">NOT NULL</span>, </span><span> c_255 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">255</span>) <span style="color: rgb(129, 0, 194);">NULL</span>, </span><span> c_1000 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">1000</span>) <span style="color: rgb(129, 0, 194);">NULL</span>, </span><span> c_4000 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">4000</span>) <span style="color: rgb(129, 0, 194);">NULL</span>, </span><span> c_8000 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">8000</span>) <span style="color: rgb(129, 0, 194);">NULL</span>, </span><span> c_16384 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">16384</span>) <span style="color: rgb(129, 0, 194);">NULL</span>, </span><span> c_65533 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">65533</span>) <span style="color: rgb(129, 0, 194);">NULL</span> </span><span>) </span><span>DUPLICATE <span style="color: rgb(129, 0, 194);">KEY</span>(id) </span><span><span style="color: rgb(129, 0, 194);">DISTRIBUTED</span> <span style="color: rgb(129, 0, 194);">BY</span> <span style="color: rgb(129, 0, 194);">HASH</span>(id) BUCKETS <span style="color: rgb(0, 128, 128);">1</span> </span><span>PROPERTIES (<span style="color: rgb(0, 128, 0);">"replication_num"</span> = <span style="color: rgb(0, 128, 0);">"3"</span>); </span><span> </span><span><span style="color: rgb(129, 0, 194);">INSERT INTO</span> t_types <span style="color: rgb(129, 0, 194);">VALUES</span> </span><span>(<span style="color: rgb(0, 128, 128);">1</span>, <span style="color: rgb(129, 0, 194);">REPEAT</span>(<span style="color: rgb(0, 128, 0);">'A'</span>,<span style="color: rgb(0, 128, 128);">255</span>), <span style="color: rgb(129, 0, 194);">REPEAT</span>(<span style="color: rgb(0, 128, 0);">'B'</span>,<span style="color: rgb(0, 128, 128);">900</span>), <span style="color: rgb(129, 0, 194);">REPEAT</span>(<span style="color: rgb(0, 128, 0);">'C'</span>,<span style="color: rgb(0, 128, 128);">900</span>), </span><span> <span style="color: rgb(129, 0, 194);">REPEAT</span>(<span style="color: rgb(0, 128, 0);">'D'</span>,<span style="color: rgb(0, 128, 128);">900</span>), <span style="color: rgb(129, 0, 194);">REPEAT</span>(<span style="color: rgb(0, 128, 0);">'E'</span>,<span style="color: rgb(0, 128, 128);">900</span>), <span style="color: rgb(129, 0, 194);">REPEAT</span>(<span style="color: rgb(0, 128, 0);">'F'</span>,<span style="color: rgb(0, 128, 128);">900</span>));</span></span></code></pre></div></div> <h3 dir="ltr">2. Inspect the protocol metadata</h3> <div role="group" aria-label="bash code" tabindex="0"><div><div></div></div><div>bash</div><div><pre style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono);"><code class="language-bash" style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono); white-space: pre;"><span><span><span style="color: rgb(0, 81, 194);">mysql</span> <span style="color: rgb(0, 128, 0);">--column-type-info</span> <span style="color: rgb(0, 128, 0);">-h</span> <<span style="color: rgb(0, 128, 0);">FE_HOS</span>T> <span style="color: rgb(0, 128, 0);">-P</span> <span style="color: rgb(0, 128, 128);">9030</span> <span style="color: rgb(0, 128, 0);">-u</span> <<span style="color: rgb(0, 128, 0);">USE</span>R> <span style="color: rgb(0, 128, 0);">-p</span> <span style="color: rgb(0, 128, 128);">\</span> </span><span> <span style="color: rgb(0, 128, 0);">-e</span> <span style="color: rgb(0, 128, 0);">"SELECT * FROM odbc_test.t_types LIMIT 1"</span></span></span></code></pre></div></div> <p dir="ltr">Observed: <code>Length: 255</code> for every string column, including <code>c_65533</code>. Expected: 1000, 4000, 8000, 16384 and 65533 respectively.</p> <h3 dir="ltr">3. Confirm the inconsistency through ODBC (optional)</h3> <div role="group" aria-label="powershell code" tabindex="0"><div><div></div></div><div>powershell</div><div><pre style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono);"><code class="language-powershell" style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono); white-space: pre;"><span><span><span style="color: rgb(179, 74, 0);">$</span>c = <span style="color: rgb(0, 81, 194);">New-Object</span> System.Data.Odbc.OdbcConnection<span style="color: rgb(43, 48, 59);">(</span><span style="color: rgb(0, 128, 0);">"DSN=<DSN>"</span><span style="color: rgb(43, 48, 59);">)</span> </span><span><span style="color: rgb(179, 74, 0);">$</span>c.Open<span style="color: rgb(43, 48, 59);">()</span> </span><span> </span><span><span style="color: rgb(110, 118, 135);"># Result set metadata -> ColumnSize 127 for every string column</span> </span><span><span style="color: rgb(179, 74, 0);">$</span>cmd = <span style="color: rgb(179, 74, 0);">$</span>c.CreateCommand<span style="color: rgb(43, 48, 59);">()</span> </span><span><span style="color: rgb(179, 74, 0);">$</span>cmd.CommandText = <span style="color: rgb(0, 128, 0);">"SELECT * FROM odbc_test.t_types LIMIT 1"</span> </span><span><span style="color: rgb(179, 74, 0);">$</span>r = <span style="color: rgb(179, 74, 0);">$</span>cmd.ExecuteReader<span style="color: rgb(43, 48, 59);">()</span> </span><span><span style="color: rgb(179, 74, 0);">$</span>schema = <span style="color: rgb(179, 74, 0);">$</span>r.GetSchemaTable<span style="color: rgb(43, 48, 59);">()</span> </span><span><span style="color: rgb(179, 74, 0);">$</span>r.Close<span style="color: rgb(43, 48, 59);">()</span> </span><span><span style="color: rgb(179, 74, 0);">$</span>schema | <span style="color: rgb(0, 81, 194);">Select-Object</span> ColumnName, ColumnSize, ProviderType | <span style="color: rgb(0, 81, 194);">Format-Table</span> </span><span> </span><span><span style="color: rgb(110, 118, 135);"># Catalog metadata -> correct declared lengths</span> </span><span><span style="color: rgb(179, 74, 0);">$</span>c.GetSchema<span style="color: rgb(43, 48, 59);">(</span><span style="color: rgb(0, 128, 0);">"Columns"</span>, <span style="color: rgb(129, 0, 194);">@</span><span style="color: rgb(43, 48, 59);">(</span><span style="color: rgb(0, 128, 0);">"odbc_test"</span>, <span style="color: rgb(179, 74, 0);">$</span><span style="color: rgb(0, 128, 128);">null</span>, <span style="color: rgb(0, 128, 0);">"t_types"</span>, <span style="color: rgb(179, 74, 0);">$</span><span style="color: rgb(0, 128, 128);">null</span><span style="color: rgb(43, 48, 59);">))</span> | </span><span> <span style="color: rgb(0, 81, 194);">Select-Object</span> COLUMN_NAME, TYPE_NAME, COLUMN_SIZE | <span style="color: rgb(0, 81, 194);">Format-Table</span> </span><span><span style="color: rgb(179, 74, 0);">$</span>c.Close<span style="color: rgb(43, 48, 59);">()</span></span></span></code></pre></div></div> <p dir="ltr">Observed <code>GetSchemaTable()</code> output:</p> <div role="group" aria-label="Code" tabindex="0"><div><div></div></div><div><pre style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono);"><code style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono); white-space: pre-wrap;">ColumnName ColumnSize ProviderType ---------- ---------- ------------ id 4 10 c_255 127 11 c_1000 127 11 c_4000 127 11 c_8000 127 11 c_16384 127 11 c_65533 127 11</code></pre></div></div> <p dir="ltr">Observed <code>GetSchema("Columns")</code> output (correct):</p> <div role="group" aria-label="Code" tabindex="0"><div><div></div></div><div><pre style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono);"><code style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono); white-space: pre-wrap;">COLUMN_NAME TYPE_NAME COLUMN_SIZE ----------- --------- ----------- c_255 varchar 255 c_1000 varchar 1000 c_4000 varchar 4000 c_8000 varchar 8000 c_16384 varchar 16384 c_65533 varchar 65533</code></pre></div></div> <p dir="ltr">Reading the values themselves through the same connection returns all 900 characters, confirming that only the metadata is affected.</p> <h3 dir="ltr">4. Control experiment against MySQL 8.0</h3> <p dir="ltr">To rule out the client and the driver, the same test was run against MySQL 8.0 (official <code>mysql:8.0</code> Docker image) using the same <code>mysql</code> CLI and the same command. The table was created with <code>DEFAULT CHARSET=latin1</code> so that the declared lengths are directly comparable in bytes, and the 65 KB column was put in a separate table to stay within the MySQL row size limit.</p> <div role="group" aria-label="sql code" tabindex="0"><div><div></div></div><div>sql</div><div><pre style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono);"><code class="language-sql" style="color: rgb(20, 24, 31); background: transparent; font-family: var(--font-mono); white-space: pre;"><span><span><span style="color: rgb(129, 0, 194);">CREATE</span> <span style="color: rgb(129, 0, 194);">TABLE</span> <span style="color: rgb(0, 81, 194);">t_types</span> ( </span><span> id <span style="color: rgb(129, 0, 194);">INT</span> <span style="color: rgb(129, 0, 194);">NOT NULL</span>, </span><span> c_255 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">255</span>), c_1000 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">1000</span>), c_4000 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">4000</span>), </span><span> c_8000 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">8000</span>), c_16384 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">16384</span>) </span><span>) <span style="color: rgb(129, 0, 194);">DEFAULT</span> CHARSET=latin1; </span><span> </span><span><span style="color: rgb(129, 0, 194);">CREATE</span> <span style="color: rgb(129, 0, 194);">TABLE</span> <span style="color: rgb(0, 81, 194);">t_big</span> ( </span><span> id <span style="color: rgb(129, 0, 194);">INT</span> <span style="color: rgb(129, 0, 194);">NOT NULL</span>, </span><span> c_65000 <span style="color: rgb(129, 0, 194);">VARCHAR</span>(<span style="color: rgb(0, 128, 128);">65000</span>) </span><span>) <span style="color: rgb(129, 0, 194);">DEFAULT</span> CHARSET=latin1;</span></span></code></pre></div></div> <p dir="ltr">MySQL 8.0 reports the declared length of each column individually:</p> <div dir="ltr"> Column | Declared | MySQL 8.0 Length | Doris 3.1.1 Length -- | -- | -- | -- c_255 | VARCHAR(255) | 255 | 255 c_1000 | VARCHAR(1000) | 1000 | 255 c_4000 | VARCHAR(4000) | 4000 | 255 c_8000 | VARCHAR(8000) | 8000 | 255 c_16384 | VARCHAR(16384) | 16384 | 255 65 KB col | see above | 65000 | 255 </div> <p dir="ltr"><code>Max_length</code> is 900 on both servers, confirming that the row data itself is transmitted correctly in either case.</p> <p dir="ltr">Two further differences appear in the same packets:</p> <ul dir="ltr"> <li>MySQL reports the actual collation of the column (<code>latin1_swedish_ci (8)</code>), while Doris always reports <code>utf8mb3_general_ci (33)</code>.</li> <li>MySQL reports the column type as <code>VAR_STRING</code>, Doris reports <code>STRING</code>.</li></ul><!--EndFragment--> </body> </html> ### Anything Else? The value appears to originate in [https://github.com/apache/doris/blob/ded08aebefdb76b167c1f5fa164feaa1b4732205/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlSerializer.java#L271](url). In getMysqlTypeLength(Type type), numeric and temporal types are handled explicitly while CHAR/VARCHAR fall through to the default branch, which carries an existing todo acknowledging that the declared field length is not used yet: java // todo:It needs to be obtained according to the field length set during the actual creation, // todo:which is not supported for the time being.default is 255 // CHAR,VARCHAR: default: return 255; The result is written into the column definition packet via writeInt4(getMysqlTypeLength(...)) in the three writeField(...) overloads. The collation in the same packet is likewise a constant, writeInt2(33), which matches the observed utf8mb3_general_ci (33). Additional notes: The server reports utf8mb3_general_ci (33) as the collation even when the client connects with charset=utf8mb4. This may be unrelated, but it affects how drivers convert the reported byte length into a character count. The server version string is reported as 5.7.99, so clients apply MySQL 5.7 compatibility behaviour when interpreting these metadata ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
