This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2689e0d7fdb [fix](exec) Return Thrift DATETIME as naive Arrow
timestamp (#67232)
2689e0d7fdb is described below
commit 2689e0d7fdb111bf822cebc26d7d5765a563e276
Author: Mryange <[email protected]>
AuthorDate: Fri Aug 28 21:10:08 2026 +0800
[fix](exec) Return Thrift DATETIME as naive Arrow timestamp (#67232)
Thrift external scanner responses encoded Doris `DATETIME` values with
the BE session timezone in the Arrow schema. This made clients interpret
wall-clock values as instants and could introduce an eight-hour offset.
The MemoryScratch sink now uses the existing naive-timestamp conversion
for `DATETIMEV2`, while preserving timezone-aware `TIMESTAMPTZ`
behavior.
### Release note
`DATETIME` values returned through the Thrift external scanner are
represented as timezone-naive Arrow timestamps.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
be/src/exec/operator/memory_scratch_sink_operator.cpp | 3 ++-
be/src/format/arrow/arrow_row_batch.cpp | 5 +++--
be/src/format/arrow/arrow_row_batch.h | 2 +-
3 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/be/src/exec/operator/memory_scratch_sink_operator.cpp
b/be/src/exec/operator/memory_scratch_sink_operator.cpp
index 5635a65168f..c59737201ec 100644
--- a/be/src/exec/operator/memory_scratch_sink_operator.cpp
+++ b/be/src/exec/operator/memory_scratch_sink_operator.cpp
@@ -106,7 +106,8 @@ Status MemoryScratchSinkOperatorX::sink_impl(RuntimeState*
state, Block* input_b
{
SCOPED_TIMER(local_state._get_arrow_schema_timer);
// After expr executed, use recaculated schema as final schema
- RETURN_IF_ERROR(get_arrow_schema_from_block(block,
&block_arrow_schema, state->timezone()));
+ RETURN_IF_ERROR(get_arrow_schema_from_block(block,
&block_arrow_schema, state->timezone(),
+ /*datetime_naive=*/true));
}
{
SCOPED_TIMER(local_state._convert_block_to_arrow_batch_timer);
diff --git a/be/src/format/arrow/arrow_row_batch.cpp
b/be/src/format/arrow/arrow_row_batch.cpp
index 4f80fb042ea..cc88bc18b92 100644
--- a/be/src/format/arrow/arrow_row_batch.cpp
+++ b/be/src/format/arrow/arrow_row_batch.cpp
@@ -203,11 +203,12 @@ std::shared_ptr<arrow::Field>
create_arrow_field_with_metadata(
}
Status get_arrow_schema_from_block(const Block& block,
std::shared_ptr<arrow::Schema>* result,
- const std::string& timezone) {
+ const std::string& timezone, bool
datetime_naive) {
std::vector<std::shared_ptr<arrow::Field>> fields;
for (const auto& type_and_name : block) {
std::shared_ptr<arrow::DataType> arrow_type;
- RETURN_IF_ERROR(convert_to_arrow_type(type_and_name.type, &arrow_type,
timezone));
+ RETURN_IF_ERROR(
+ convert_to_arrow_type(type_and_name.type, &arrow_type,
timezone, datetime_naive));
auto field = create_arrow_field_with_metadata(type_and_name.name,
arrow_type,
type_and_name.type->is_nullable(),
type_and_name.type->get_primitive_type());
diff --git a/be/src/format/arrow/arrow_row_batch.h
b/be/src/format/arrow/arrow_row_batch.h
index e5ddd18fae6..d5ba5cb0ed0 100644
--- a/be/src/format/arrow/arrow_row_batch.h
+++ b/be/src/format/arrow/arrow_row_batch.h
@@ -54,7 +54,7 @@ std::shared_ptr<arrow::Field>
create_arrow_field_with_metadata(
bool is_nullable, PrimitiveType primitive_type);
Status get_arrow_schema_from_block(const Block& block,
std::shared_ptr<arrow::Schema>* result,
- const std::string& timezone);
+ const std::string& timezone, bool
datetime_naive = false);
Status get_arrow_schema_from_expr_ctxs(const VExprContextSPtrs&
output_vexpr_ctxs,
std::shared_ptr<arrow::Schema>* result,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
