Doris-Breakwater commented on issue #68392: URL: https://github.com/apache/doris/issues/68392#issuecomment-5776603195
Thanks for the unusually precise report. The code and the numeric result support a specific compatibility regression. ### Initial assessment **Verified from the public sources:** this is the interaction of an older Doris Thrift/Arrow schema bug with the stricter timestamp decoding introduced in Flink Connector 26.2.0. It is not consistent with a Flink result-viewer-only or `CAST(... AS STRING)` problem. The server-side chain in the Apache Doris 2.1.8 source is: 1. [`DataTypeDateTimeV2::get_type_as_type_descriptor()`](https://github.com/apache/doris/blob/2.1.8-rc01/be/src/vec/data_types/data_type_time_v2.h#L114-L118) discards the column's real scale and creates a default `TYPE_DATETIMEV2` descriptor. That descriptor [defaults to scale 6](https://github.com/apache/doris/blob/2.1.8-rc01/be/src/runtime/types.h#L67-L77), so the generated Arrow field advertises `MICROSECOND` even for `DATETIMEV2(0)`. 2. The value serializer still uses the real SerDe scale: [scale 0 emits epoch seconds, scales 1-3 emit milliseconds, and scales 4-6 emit microseconds](https://github.com/apache/doris/blob/2.1.8-rc01/be/src/vec/data_types/serde/data_type_datetimev2_serde.cpp#L98-L121). Thus the schema and value unit disagree for scales 0-3. 3. Connector 25.0.0 hides that invalid wire representation by inferring the unit from numeric magnitude. [Connector PR #676](https://github.com/apache/doris-flink-connector/pull/676) removed the heuristic and began honoring Arrow unit metadata; the first released tag containing that change is 26.2.0, so 26.2.0 and 26.3.0 expose the server-side mismatch. 26.1.1 and earlier retain the heuristic. The reported value is an exact fingerprint of this path: `2026-07-30 19:44:28 +08:00` is epoch second `1785411868`; decoding `1785411868` as microseconds gives `1970-01-01 08:29:45.411868 +08:00`. Apache fixed the lost scale on branch 2.1 in [PR #49244](https://github.com/apache/doris/pull/49244) by copying `_scale` into the Arrow type descriptor. In the public source tags I checked, `2.1.8-rc01` and `2.1.9-rc01` lack the fix, while `2.1.9-rc02`, `2.1.10-rc01`, and `2.1.11-rc01` contain it. This establishes that the stock Apache Doris 2.1.8 source has the same mechanism, although the report has not run a stock 2.1.8 binary reproduction. ### Scope and compatibility - The code-backed affected combination is the **Thrift source path**, a server build lacking the scale-preservation fix, `DATETIMEV2` scale 0-3, and Connector 26.2.0 or newer. - Scale 4-6 already uses microsecond values, so it is not affected by this specific unit mismatch. - This report does not exercise Flight SQL, and its producer path has changed separately; I would not declare Flight SQL affected from this evidence. - `SelectDB 2.1.8-3` is a distribution-specific version. Its exact mapping cannot be inferred from the name. The observed value strongly indicates that its BE lacks the equivalent fix, but the exact build revision and raw Arrow field still need to be captured before treating that deployment detail as confirmed. ### Recommended action 1. **Preferred fix:** upgrade the server/BE to a build containing the equivalent of Apache Doris PR #49244 (preserve `DataTypeDateTimeV2::_scale` when constructing the Arrow descriptor). FE and BE build revisions should both be checked; the BE emits the Arrow batch. 2. **Temporary workaround:** keep the already validated Connector 25.0.0 pin until the server is upgraded. Connector 26.1.1 is the last release before PR #676 and has the same magnitude-based compatibility behavior, but it has not been validated in this environment, so it should not be presented as confirmed without a test. 3. I found no Connector 26.3.0 option that restores legacy magnitude decoding. Automatically reintroducing that heuristic is unsafe because magnitude is not an unambiguous timestamp-unit signal. If compatibility must be connector-side, an explicit legacy mode or a server capability/version handshake would be safer. 4. The connector compatibility documentation should state that 26.2.0+ requires a Doris/SelectDB build with correct Arrow `DATETIMEV2` unit metadata, and should reference the server-side scale-preservation fix rather than only giving a broad Doris version range. This is primarily an interoperability issue for `apache/doris-flink-connector` (the compatibility change is PR #676), while the producer-side remediation for this particular server build belongs with the SelectDB distribution. A regression test matrix should cover server-before/server-after PR #49244 against connector 26.1.1/26.2.0+, with `DATETIMEV2(0)`, `(3)`, and `(6)`. ### Diagnostics still needed to confirm the reported deployment Please provide sanitized versions of: - full FE and BE version/build commit output (not only `2.1.8-3`) and `SHOW CREATE TABLE` for the real source columns, including their scales; - for one affected Arrow cell: `fieldVector.getClass().getName()`, `fieldVector.getField().getType()` (unit and timezone), and `((TimeStampVector) fieldVector).get(rowIndex)`; the predicted tuple is a microsecond-declared timestamp carrying raw value `1785411868`; - the loaded `RowBatch` class location (for example, `RowBatch.class.getProtectionDomain().getCodeSource().getLocation()`) plus the deployed connector JAR checksum or dependency tree, to rule out a classpath collision; - if practical, the same minimal table/row tested against a stock Apache Doris build before and after PR #49244. No Doris execution profile is needed for this correctness issue. The Arrow field metadata and raw long value are the decisive evidence. Breakwater-GitHub-Analysis-Slot: slot_bd1f9dd2153c -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
