Doris-Breakwater commented on issue #69041: URL: https://github.com/apache/doris/issues/69041#issuecomment-6150879101
The timezone-aware reproduction is supported by the source: this is a BE Arrow Stream Load conversion bug, with high confidence in the identified mechanism. The proposed reader change addresses that mechanism. There is also a version distinction for timezone-naive timestamps that should be clarified before backporting. I checked the local `4.1.3` (`7126cf65d96e`) and `4.1.4` (`ad35a140c7fd`) tags, and fetched current master at `1f3cd9092fbc9151a3cc038d36e32aef68bd8357`. This assessment is based on source inspection; I have not run the reproduction against a Doris cluster or executed the proposed patch's tests. The issue currently has no labels; BE ingestion/data correctness is the appropriate triage area. The verified path is: - Both header spellings are parsed: `timezone` takes precedence over `time_zone`. The request timezone is validated by the load task, written into `TQueryGlobals.time_zone`, and resolved into `RuntimeState::timezone_obj()`. Thus the explicit UTC setting reaches runtime state. See the [HTTP handler](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/service/http/action/stream_load.cpp#L528-L533), [load task](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsStreamLoadTask.java#L435-L440), [planner](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsStreamLoadPlanner.java#L326-L329), and [runtime initialization](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/runtime/runtime_state.cpp#L177-L203). - `FileScanner` supplies that state to `ArrowStreamReader`, but the reader initializes `_ctzz` independently from `default_time_zone`, then passes `_ctzz` into the column SerDe. The constant is actually fixed `+08:00`, rather than the named `Asia/Shanghai` zone. Parquet receives the runtime timezone directly. See the [scanner](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/exec/scan/file_scanner.cpp#L1146-L1281), [current Arrow reader](https://github.com/apache/doris/blob/1f3cd9092fbc9151a3cc038d36e32aef68bd8357/be/src/format/arrow/arrow_stream_reader.cpp#L42-L120), and [default constant](https://github.com/apache/doris/blob/1f3cd9092fbc9151a3cc038d36e32aef68bd8357/be/src/util/timezone_utils.cpp#L60). - For a timezone-aware Arrow timestamp, DATETIMEV2 conversion uses the supplied zone to turn epoch seconds into stored date/time fields. This explains the reported next-day `00:37:00` from a UTC instant of `16:37:00`; changing the SELECT session timezone does not repair those stored fields. The statement that naive `timestamp[us]` behaves identically needs a version qualification. In [4.1.3](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp#L397-L445), decoding always uses the supplied zone. In [4.1.4](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp#L622-L647) and [current master](https://github.com/apache/doris/blob/1f3cd9092fbc9151a3cc038d36e32aef68bd8357/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp#L699-L730), an empty Arrow timezone explicitly selects UTC to preserve naive wall-clock fields. Public [PR #66489](https://github.com/apache/doris/pull/66489) contains the related SerDe correction, but does not change `ArrowStreamReader`; it does not resolve the timezone-aware load-header bug. Please open the offered PR with `ArrowStreamReaderStreamingTest`. Using `state->timezone_obj()` and retaining the existing default for a null state is consistent with the scanner contract. Acceptance coverage should exercise the actual IPC reader with an initialized runtime state, rather than only calling the SerDe: - `timestamp[us, tz=UTC]`: UTC load stores `2026-10-13 16:37:00`; Asia/Shanghai load stores `2026-10-14 00:37:00`. - Naive `timestamp[us]`: preserve `2026-10-13 16:37:00` under both zones on 4.1.4/master. Preserve existing fractional-second, pre-epoch, and nullable-column coverage, and test the constructor's null-state fallback. - Add an HTTP regression to verify header propagation, including both spellings and the no-header default. For a 4.1.3 backport, account for the older naive SerDe behavior; the constructor change alone would still shift naive timestamps under a non-UTC load timezone. The remaining evidence needed is the PR/test output and a minimal Arrow-generation/load script with PyArrow version, emitted schema, raw integer timestamp values, exact FE/BE build identifiers, and load responses plus SELECT results for each claimed version. These would resolve the naive-timestamp discrepancy. No Profile is needed for this correctness issue; BE logs are useful if those results still diverge from the verified path. For now, the Parquet route already demonstrated with `timezone: UTC` is a practical workaround for the timezone-aware example. Document the fix as a change in newly loaded DATETIME values according to the effective load timezone. An explicit `timezone: Asia/Shanghai` preserves the old modern-date +8-hour result for timezone-aware input after the fix. Existing stored values remain unchanged and need separate validation/reload; avoid adding a blanket eight-hour subtraction that could double-correct data later. Breakwater-GitHub-Analysis-Slot: slot_90d1d24f1c4f -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
