github-actions[bot] commented on code in PR #66333:
URL: https://github.com/apache/doris/pull/66333#discussion_r3713578184
##########
be/src/core/data_type/data_type_date_or_datetime_v2.h:
##########
@@ -154,10 +160,56 @@ class DataTypeDateTimeV2 final : public
DataTypeNumberBase<PrimitiveType::TYPE_D
UInt32 _scale;
};
+class DataTypeDateTimeV2Nano final
+ : public DataTypeNumberBase<PrimitiveType::TYPE_DATETIMEV2_NANO> {
+public:
+ static constexpr PrimitiveType PType = TYPE_DATETIMEV2_NANO;
+ static constexpr bool is_parametric = true;
+
+ explicit DataTypeDateTimeV2Nano(UInt32 scale = 9) : _scale(scale) {
+ DORIS_CHECK_GE(scale, 7);
+ DORIS_CHECK_LE(scale, 9);
+ }
+
+ PrimitiveType get_primitive_type() const override { return
TYPE_DATETIMEV2_NANO; }
+ doris::FieldType get_storage_field_type() const override {
+ return doris::FieldType::OLAP_FIELD_TYPE_DATETIMEV2_NANO;
+ }
+ void to_protobuf(PTypeDesc* ptype, PTypeNode* node, PScalarType*
scalar_type) const override {
+ scalar_type->set_scale(_scale);
+ }
+
+ const std::string get_family_name() const override { return
"DateTimeV2Nano"; }
Review Comment:
[P1] Register runtime functions for the new family
Giving scale-7..9 values a distinct `DateTimeV2Nano` family changes BE
function lookup keys, but the FE still accepts these values for wildcard
DATETIMEV2 functions. For example, a stored `dt9` makes `date_format(dt9,
'%Y')` request `date_formatDateTimeV2NanoString`, while only the legacy-family
key is registered; `convert_tz` similarly reaches an implementation that
asserts `ColumnDateTimeV2`. The added cases use foldable literals, so they do
not exercise this column path. Please register/implement the nano variants (and
cover stored columns) before exposing this family.
##########
be/src/exprs/function/cast/function_cast_date.cpp:
##########
@@ -30,9 +30,14 @@ WrapperType create_datelike_wrapper(FunctionContext*
context, const DataTypePtr&
auto make_datelike_wrapper = [&](const auto& types) -> bool {
using Types = std::decay_t<decltype(types)>;
using FromDataType = typename Types::LeftType;
- if constexpr (CastUtil::IsPureDigitType<FromDataType> ||
IsDatelikeTypes<FromDataType> ||
- IsStringType<FromDataType> ||
- std::is_same_v<FromDataType, DataTypeTimeStampTz>) {
+ constexpr bool is_nano_source = std::is_same_v<FromDataType,
DataTypeDateTimeV2Nano>;
+ constexpr bool is_nano_target = std::is_same_v<ToDataType,
DataTypeDateTimeV2Nano>;
+ constexpr bool is_supported_nano_cast = is_nano_target &&
IsStringType<FromDataType>;
Review Comment:
[P1] Support non-identity casts involving nano datetimes
This accepts only string-to-nano, while the legacy branch rejects every nano
endpoint. Consequently stored-column casts fail not only across the scale-6/7
representation boundary in both directions, but also within the nano family:
`DATETIMEV2(9)` to `DATETIMEV2(8)` is non-identity because equality includes
scale and then reaches the unsupported wrapper. FE can insert the cross-family
failure implicitly for comparisons, joins, or unions. Please implement all
required scale conversions and add non-foldable column tests.
##########
be/src/core/data_type_serde/data_type_datetimev2_nano_serde.cpp:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "core/data_type_serde/data_type_datetimev2_nano_serde.h"
+
+#include <arrow/array.h>
+#include <arrow/builder.h>
+#include <cctz/time_zone.h>
+
+#include <cctype>
+#include <limits>
+#include <orc/Vector.hh>
+#include <string>
+
+#include "common/config.h"
+#include "core/assert_cast.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/data_type_serde/arrow_validation.h"
+#include "core/data_type_serde/decoded_column_view.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/function/cast/cast_to_datetimev2_impl.hpp"
+#include "util/mysql_row_buffer.h"
+#include "util/unaligned.h"
+
+namespace doris {
+namespace {
+
+constexpr int64_t NANOS_PER_MILLISECOND = 1000000;
+constexpr int64_t NANOS_PER_MICROSECOND = 1000;
+
+bool checked_scale_to_nanos(int64_t value, int64_t multiplier, int64_t*
result) {
+ return !__builtin_mul_overflow(value, multiplier, result);
+}
+
+Status utc_epoch_nanos_to_local_epoch_nanos(int64_t source, const
cctz::time_zone& timezone,
+ int64_t* result) {
+ const DateTimeV2NanoValue source_value(source);
+ DateV2Value<DateTimeV2ValueType> local_value;
+ local_value.from_unixtime(source_value.epoch_seconds(), timezone);
+ local_value.set_microsecond(source_value.microsecond());
+ DateTimeV2NanoValue target_value;
+ if (!target_value.from_datetime(local_value,
source_value.nanosecond_remainder())) {
+ return Status::DataQualityError("Timestamp {} is outside DATETIMEV2
nanosecond range",
+ source);
+ }
+ *result = target_value.epoch_nanos();
+ return Status::OK();
+}
+
+Status local_epoch_nanos_to_utc_epoch_nanos(int64_t source, const
cctz::time_zone& timezone,
+ int64_t* result) {
+ const DateTimeV2NanoValue source_value(source);
+ auto local_value = source_value.to_datetime();
+ int64_t seconds = 0;
+ local_value.unix_timestamp(&seconds, timezone);
+ const __int128 epoch_nanos =
+ static_cast<__int128>(seconds) *
DateTimeV2NanoValue::NANOS_PER_SECOND +
+ source_value.nanosecond();
+ if (epoch_nanos < std::numeric_limits<int64_t>::min() ||
+ epoch_nanos > std::numeric_limits<int64_t>::max()) {
+ return Status::DataQualityError("DATETIMEV2 value {} is outside epoch
nanosecond range",
+ source_value.to_string(9));
+ }
+ *result = static_cast<int64_t>(epoch_nanos);
+ return Status::OK();
+}
+
+} // namespace
+
+Status parse_datetimev2_nano(StringRef str, int scale, int64_t* epoch_nanos,
+ const cctz::time_zone* local_time_zone) {
+ DORIS_CHECK_GE(scale, 7);
+ DORIS_CHECK_LE(scale, 9);
+
+ std::string input(str.data, str.size);
+ size_t fraction_begin = std::string::npos;
+ size_t fraction_end = std::string::npos;
+ const size_t dot = input.rfind('.');
+ if (dot != std::string::npos && dot + 1 < input.size() &&
+ std::isdigit(static_cast<unsigned char>(input[dot + 1]))) {
+ fraction_begin = dot + 1;
+ fraction_end = fraction_begin;
+ while (fraction_end < input.size() &&
+ std::isdigit(static_cast<unsigned char>(input[fraction_end]))) {
+ ++fraction_end;
+ }
+ }
+
+ uint32_t nanos = 0;
+ size_t fraction_length = 0;
+ if (fraction_begin != std::string::npos) {
+ fraction_length = fraction_end - fraction_begin;
+ const size_t copied_digits = std::min<size_t>(fraction_length, 9);
+ for (size_t i = 0; i < copied_digits; ++i) {
+ nanos = nanos * 10 + static_cast<uint32_t>(input[fraction_begin +
i] - '0');
+ }
+ for (size_t i = copied_digits; i < 9; ++i) {
+ nanos *= 10;
+ }
+ }
+
+ const auto quantum = static_cast<uint32_t>(int_exp10(9 - scale));
+ nanos = nanos / quantum * quantum;
+ if (fraction_length > static_cast<size_t>(scale) && input[fraction_begin +
scale] >= '5') {
+ nanos += quantum;
+ }
+
+ std::string base = input;
+ if (fraction_begin != std::string::npos) {
+ base.erase(dot, fraction_end - dot);
+ }
+ const StringRef base_ref(base.data(), base.size());
+ DateV2Value<DateTimeV2ValueType> datetime;
+ CastParameters params {.status = Status::OK(), .is_strict = true};
+ CastToDatetimeV2::from_string_strict_mode<DatelikeParseMode::STRICT>(
+ base_ref, datetime, local_time_zone, 0, params);
+ if (!params.status.ok()) {
+ return params.status;
+ }
+
+ if (nanos == DateTimeV2NanoValue::NANOS_PER_SECOND) {
+ if (!datetime.date_add_interval<TimeUnit::SECOND>(
+ TimeInterval {TimeUnit::SECOND, 1, false})) {
+ return Status::InvalidArgument("DATETIMEV2 value overflows while
rounding '{}'", input);
+ }
+ nanos = 0;
+ }
+ datetime.set_microsecond(nanos / NANOS_PER_MICROSECOND);
+ DateTimeV2NanoValue value;
+ if (!value.from_datetime(datetime, static_cast<uint16_t>(nanos %
NANOS_PER_MICROSECOND))) {
+ return Status::InvalidArgument(
+ "DATETIMEV2({}) value '{}' is outside [{}, {}]", scale, input,
+
DateTimeV2NanoValue(std::numeric_limits<int64_t>::min()).to_string(scale),
+
DateTimeV2NanoValue(std::numeric_limits<int64_t>::max()).to_string(scale));
+ }
+ *epoch_nanos = value.epoch_nanos();
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_batch(const ColumnString&
strings,
+ ColumnNullable& result,
+ const FormatOptions&
options) const {
+ auto& data =
assert_cast<ColumnDateTimeV2Nano&>(result.get_nested_column()).get_data();
+ auto& null_map = result.get_null_map_column().get_data();
+ result.resize(strings.size());
+ for (size_t i = 0; i < strings.size(); ++i) {
+ int64_t value = 0;
+ const auto status =
+ parse_datetimev2_nano(strings.get_data_at(i), _scale, &value,
options.timezone);
+ null_map[i] = !status.ok();
+ data[i] = DateTimeV2NanoValue(value);
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_strict_mode_batch(
+ const ColumnString& strings, IColumn& result, const FormatOptions&
options,
+ const NullMap::value_type* null_map) const {
+ auto& data = assert_cast<ColumnDateTimeV2Nano&>(result).get_data();
+ result.resize(strings.size());
+ for (size_t i = 0; i < strings.size(); ++i) {
+ if (null_map != nullptr && null_map[i]) {
+ continue;
+ }
+ int64_t value = 0;
+ RETURN_IF_ERROR(
+ parse_datetimev2_nano(strings.get_data_at(i), _scale, &value,
options.timezone));
+ data[i] = DateTimeV2NanoValue(value);
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string(StringRef& str, IColumn&
column,
+ const FormatOptions& options)
const {
+ int64_t value = 0;
+ RETURN_IF_ERROR(parse_datetimev2_nano(str, _scale, &value,
options.timezone));
+
assert_cast<ColumnDateTimeV2Nano&>(column).insert_value(DateTimeV2NanoValue(value));
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_strict_mode(StringRef& str,
IColumn& column,
+ const
FormatOptions& options) const {
+ return from_string(str, column, options);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::serialize_column_to_json(const IColumn&
column,
+ int64_t
start_idx, int64_t end_idx,
+ BufferWritable&
bw,
+ FormatOptions&
options) const {
+ SERIALIZE_COLUMN_TO_JSON();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::serialize_one_cell_to_json(const IColumn&
column,
+ int64_t
row_num, BufferWritable& bw,
+ FormatOptions&
options) const {
+ auto [column_ptr, index] = check_column_const_set_readability(column,
row_num);
+ if (_nesting_level > 1) {
+ bw.write('"');
+ }
+ const auto value =
+ assert_cast<const ColumnDateTimeV2Nano&,
TypeCheckOnRelease::DISABLE>(*column_ptr)
+ .get_element(index);
+ const std::string result = value.to_string(_scale);
+ bw.write(result.data(), result.size());
+ if (_nesting_level > 1) {
+ bw.write('"');
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::deserialize_column_from_json_vector(
+ IColumn& column, std::vector<Slice>& slices, uint64_t*
num_deserialized,
+ const FormatOptions& options) const {
+ DESERIALIZE_COLUMN_FROM_JSON_VECTOR();
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::deserialize_one_cell_from_json(
+ IColumn& column, Slice& slice, const FormatOptions& options) const {
+ if (_nesting_level > 1) {
+ slice.trim_quote();
+ }
+ StringRef str(slice.data, slice.size);
+ return from_string(str, column, options);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::write_column_to_arrow(const IColumn&
column,
+ const NullMap*
null_map,
+ arrow::ArrayBuilder*
array_builder,
+ int64_t start,
int64_t end,
+ const
cctz::time_zone& ctz) const {
+ const auto& data = assert_cast<const
ColumnDateTimeV2Nano&>(column).get_data();
+ auto& builder = assert_cast<arrow::TimestampBuilder&>(*array_builder);
+ const auto timestamp_type =
std::static_pointer_cast<arrow::TimestampType>(builder.type());
+ const auto& timezone = timestamp_type->timezone();
+ for (int64_t i = start; i < end; ++i) {
+ if (null_map != nullptr && (*null_map)[i]) {
+ RETURN_IF_ERROR(checkArrowStatus(builder.AppendNull(), column,
builder));
+ continue;
+ }
+ int64_t value = data[i].epoch_nanos();
+ if (!timezone.empty()) {
+ RETURN_IF_ERROR(local_epoch_nanos_to_utc_epoch_nanos(value, ctz,
&value));
+ }
+ switch (timestamp_type->unit()) {
+ case arrow::TimeUnit::SECOND:
+ value /= DateTimeV2NanoValue::NANOS_PER_SECOND;
Review Comment:
[P1] Floor pre-epoch values when downscaling Arrow timestamps
C++ signed division truncates toward zero here, so `-1ns` is emitted as
Arrow second `0` instead of `-1`; the MILLI and MICRO branches have the same
problem. Doris's own `epoch_seconds()` and Parquet nanos-to-micros conversion
decrement a negative quotient when a remainder is discarded. Please apply that
rule to every coarser Arrow unit and cover negative non-aligned values.
##########
be/src/storage/tablet/tablet_schema.cpp:
##########
@@ -478,6 +482,9 @@ void TabletColumn::init_from_pb(const ColumnPB& column) {
if (column.has_frac()) {
_frac = column.frac();
}
+ if (_type == FieldType::OLAP_FIELD_TYPE_DATETIMEV2 && _frac > 6) {
+ _type = FieldType::OLAP_FIELD_TYPE_DATETIMEV2_NANO;
Review Comment:
[P1] Extend delete predicates for the new storage type
After this rewrite, a scale-7..9 column reaches `DeleteHandler` as
`OLAP_FIELD_TYPE_DATETIMEV2_NANO`, but condition validation handles only the
legacy datetime field type and rejects the new enum as unknown. The IN/NOT IN
replay converter likewise has no `TYPE_DATETIMEV2_NANO` case. As a result,
DELETE predicates on these columns fail even though scan predicates work.
Please add validation/parsing for both comparison and list predicates and cover
an actual DELETE.
##########
be/src/core/data_type_serde/data_type_datetimev2_nano_serde.cpp:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "core/data_type_serde/data_type_datetimev2_nano_serde.h"
+
+#include <arrow/array.h>
+#include <arrow/builder.h>
+#include <cctz/time_zone.h>
+
+#include <cctype>
+#include <limits>
+#include <orc/Vector.hh>
+#include <string>
+
+#include "common/config.h"
+#include "core/assert_cast.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/data_type_serde/arrow_validation.h"
+#include "core/data_type_serde/decoded_column_view.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/function/cast/cast_to_datetimev2_impl.hpp"
+#include "util/mysql_row_buffer.h"
+#include "util/unaligned.h"
+
+namespace doris {
+namespace {
+
+constexpr int64_t NANOS_PER_MILLISECOND = 1000000;
+constexpr int64_t NANOS_PER_MICROSECOND = 1000;
+
+bool checked_scale_to_nanos(int64_t value, int64_t multiplier, int64_t*
result) {
+ return !__builtin_mul_overflow(value, multiplier, result);
+}
+
+Status utc_epoch_nanos_to_local_epoch_nanos(int64_t source, const
cctz::time_zone& timezone,
+ int64_t* result) {
+ const DateTimeV2NanoValue source_value(source);
+ DateV2Value<DateTimeV2ValueType> local_value;
+ local_value.from_unixtime(source_value.epoch_seconds(), timezone);
+ local_value.set_microsecond(source_value.microsecond());
+ DateTimeV2NanoValue target_value;
+ if (!target_value.from_datetime(local_value,
source_value.nanosecond_remainder())) {
+ return Status::DataQualityError("Timestamp {} is outside DATETIMEV2
nanosecond range",
+ source);
+ }
+ *result = target_value.epoch_nanos();
+ return Status::OK();
+}
+
+Status local_epoch_nanos_to_utc_epoch_nanos(int64_t source, const
cctz::time_zone& timezone,
+ int64_t* result) {
+ const DateTimeV2NanoValue source_value(source);
+ auto local_value = source_value.to_datetime();
+ int64_t seconds = 0;
+ local_value.unix_timestamp(&seconds, timezone);
+ const __int128 epoch_nanos =
+ static_cast<__int128>(seconds) *
DateTimeV2NanoValue::NANOS_PER_SECOND +
+ source_value.nanosecond();
+ if (epoch_nanos < std::numeric_limits<int64_t>::min() ||
+ epoch_nanos > std::numeric_limits<int64_t>::max()) {
+ return Status::DataQualityError("DATETIMEV2 value {} is outside epoch
nanosecond range",
+ source_value.to_string(9));
+ }
+ *result = static_cast<int64_t>(epoch_nanos);
+ return Status::OK();
+}
+
+} // namespace
+
+Status parse_datetimev2_nano(StringRef str, int scale, int64_t* epoch_nanos,
+ const cctz::time_zone* local_time_zone) {
+ DORIS_CHECK_GE(scale, 7);
+ DORIS_CHECK_LE(scale, 9);
+
+ std::string input(str.data, str.size);
+ size_t fraction_begin = std::string::npos;
+ size_t fraction_end = std::string::npos;
+ const size_t dot = input.rfind('.');
+ if (dot != std::string::npos && dot + 1 < input.size() &&
+ std::isdigit(static_cast<unsigned char>(input[dot + 1]))) {
+ fraction_begin = dot + 1;
+ fraction_end = fraction_begin;
+ while (fraction_end < input.size() &&
+ std::isdigit(static_cast<unsigned char>(input[fraction_end]))) {
+ ++fraction_end;
+ }
+ }
+
+ uint32_t nanos = 0;
+ size_t fraction_length = 0;
+ if (fraction_begin != std::string::npos) {
+ fraction_length = fraction_end - fraction_begin;
+ const size_t copied_digits = std::min<size_t>(fraction_length, 9);
+ for (size_t i = 0; i < copied_digits; ++i) {
+ nanos = nanos * 10 + static_cast<uint32_t>(input[fraction_begin +
i] - '0');
+ }
+ for (size_t i = copied_digits; i < 9; ++i) {
+ nanos *= 10;
+ }
+ }
+
+ const auto quantum = static_cast<uint32_t>(int_exp10(9 - scale));
+ nanos = nanos / quantum * quantum;
+ if (fraction_length > static_cast<size_t>(scale) && input[fraction_begin +
scale] >= '5') {
+ nanos += quantum;
+ }
+
+ std::string base = input;
+ if (fraction_begin != std::string::npos) {
+ base.erase(dot, fraction_end - dot);
+ }
+ const StringRef base_ref(base.data(), base.size());
+ DateV2Value<DateTimeV2ValueType> datetime;
+ CastParameters params {.status = Status::OK(), .is_strict = true};
+ CastToDatetimeV2::from_string_strict_mode<DatelikeParseMode::STRICT>(
+ base_ref, datetime, local_time_zone, 0, params);
+ if (!params.status.ok()) {
+ return params.status;
+ }
+
+ if (nanos == DateTimeV2NanoValue::NANOS_PER_SECOND) {
+ if (!datetime.date_add_interval<TimeUnit::SECOND>(
+ TimeInterval {TimeUnit::SECOND, 1, false})) {
+ return Status::InvalidArgument("DATETIMEV2 value overflows while
rounding '{}'", input);
+ }
+ nanos = 0;
+ }
+ datetime.set_microsecond(nanos / NANOS_PER_MICROSECOND);
+ DateTimeV2NanoValue value;
+ if (!value.from_datetime(datetime, static_cast<uint16_t>(nanos %
NANOS_PER_MICROSECOND))) {
+ return Status::InvalidArgument(
+ "DATETIMEV2({}) value '{}' is outside [{}, {}]", scale, input,
+
DateTimeV2NanoValue(std::numeric_limits<int64_t>::min()).to_string(scale),
+
DateTimeV2NanoValue(std::numeric_limits<int64_t>::max()).to_string(scale));
+ }
+ *epoch_nanos = value.epoch_nanos();
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_batch(const ColumnString&
strings,
+ ColumnNullable& result,
+ const FormatOptions&
options) const {
+ auto& data =
assert_cast<ColumnDateTimeV2Nano&>(result.get_nested_column()).get_data();
+ auto& null_map = result.get_null_map_column().get_data();
+ result.resize(strings.size());
+ for (size_t i = 0; i < strings.size(); ++i) {
+ int64_t value = 0;
+ const auto status =
+ parse_datetimev2_nano(strings.get_data_at(i), _scale, &value,
options.timezone);
+ null_map[i] = !status.ok();
+ data[i] = DateTimeV2NanoValue(value);
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_strict_mode_batch(
+ const ColumnString& strings, IColumn& result, const FormatOptions&
options,
+ const NullMap::value_type* null_map) const {
+ auto& data = assert_cast<ColumnDateTimeV2Nano&>(result).get_data();
+ result.resize(strings.size());
+ for (size_t i = 0; i < strings.size(); ++i) {
+ if (null_map != nullptr && null_map[i]) {
+ continue;
+ }
+ int64_t value = 0;
+ RETURN_IF_ERROR(
+ parse_datetimev2_nano(strings.get_data_at(i), _scale, &value,
options.timezone));
+ data[i] = DateTimeV2NanoValue(value);
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string(StringRef& str, IColumn&
column,
+ const FormatOptions& options)
const {
+ int64_t value = 0;
+ RETURN_IF_ERROR(parse_datetimev2_nano(str, _scale, &value,
options.timezone));
+
assert_cast<ColumnDateTimeV2Nano&>(column).insert_value(DateTimeV2NanoValue(value));
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_strict_mode(StringRef& str,
IColumn& column,
+ const
FormatOptions& options) const {
+ return from_string(str, column, options);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::serialize_column_to_json(const IColumn&
column,
+ int64_t
start_idx, int64_t end_idx,
+ BufferWritable&
bw,
+ FormatOptions&
options) const {
+ SERIALIZE_COLUMN_TO_JSON();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::serialize_one_cell_to_json(const IColumn&
column,
+ int64_t
row_num, BufferWritable& bw,
+ FormatOptions&
options) const {
+ auto [column_ptr, index] = check_column_const_set_readability(column,
row_num);
+ if (_nesting_level > 1) {
+ bw.write('"');
+ }
+ const auto value =
+ assert_cast<const ColumnDateTimeV2Nano&,
TypeCheckOnRelease::DISABLE>(*column_ptr)
+ .get_element(index);
+ const std::string result = value.to_string(_scale);
+ bw.write(result.data(), result.size());
+ if (_nesting_level > 1) {
+ bw.write('"');
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::deserialize_column_from_json_vector(
+ IColumn& column, std::vector<Slice>& slices, uint64_t*
num_deserialized,
+ const FormatOptions& options) const {
+ DESERIALIZE_COLUMN_FROM_JSON_VECTOR();
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::deserialize_one_cell_from_json(
+ IColumn& column, Slice& slice, const FormatOptions& options) const {
+ if (_nesting_level > 1) {
+ slice.trim_quote();
+ }
+ StringRef str(slice.data, slice.size);
+ return from_string(str, column, options);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::write_column_to_arrow(const IColumn&
column,
+ const NullMap*
null_map,
+ arrow::ArrayBuilder*
array_builder,
+ int64_t start,
int64_t end,
+ const
cctz::time_zone& ctz) const {
+ const auto& data = assert_cast<const
ColumnDateTimeV2Nano&>(column).get_data();
+ auto& builder = assert_cast<arrow::TimestampBuilder&>(*array_builder);
+ const auto timestamp_type =
std::static_pointer_cast<arrow::TimestampType>(builder.type());
+ const auto& timezone = timestamp_type->timezone();
+ for (int64_t i = start; i < end; ++i) {
+ if (null_map != nullptr && (*null_map)[i]) {
+ RETURN_IF_ERROR(checkArrowStatus(builder.AppendNull(), column,
builder));
+ continue;
+ }
+ int64_t value = data[i].epoch_nanos();
+ if (!timezone.empty()) {
+ RETURN_IF_ERROR(local_epoch_nanos_to_utc_epoch_nanos(value, ctz,
&value));
+ }
+ switch (timestamp_type->unit()) {
+ case arrow::TimeUnit::SECOND:
+ value /= DateTimeV2NanoValue::NANOS_PER_SECOND;
+ break;
+ case arrow::TimeUnit::MILLI:
+ value /= NANOS_PER_MILLISECOND;
+ break;
+ case arrow::TimeUnit::MICRO:
+ value /= NANOS_PER_MICROSECOND;
+ break;
+ case arrow::TimeUnit::NANO:
+ break;
+ }
+ RETURN_IF_ERROR(checkArrowStatus(builder.Append(value), column,
builder));
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::read_column_from_arrow(IColumn& column,
+ const arrow::Array*
arrow_array,
+ int64_t start,
int64_t end,
+ const
cctz::time_zone& ctz) const {
+ if (arrow_array->type_id() != arrow::Type::TIMESTAMP) {
+ return Status::InvalidArgument("Expected Arrow timestamp, got {}",
arrow_array->type_id());
+ }
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
+ const auto& array = assert_cast<const
arrow::TimestampArray&>(*arrow_array);
+ const auto type =
std::static_pointer_cast<arrow::TimestampType>(array.type());
+ auto& data = assert_cast<ColumnDateTimeV2Nano&>(column).get_data();
+ for (int64_t i = start; i < end; ++i) {
+ int64_t value = array.Value(i);
+ int64_t nanos = 0;
+ switch (type->unit()) {
+ case arrow::TimeUnit::SECOND:
+ if (!checked_scale_to_nanos(value,
DateTimeV2NanoValue::NANOS_PER_SECOND, &nanos)) {
+ return Status::DataQualityError("Arrow timestamp {} overflows
nanoseconds", value);
+ }
+ break;
+ case arrow::TimeUnit::MILLI:
+ if (!checked_scale_to_nanos(value, NANOS_PER_MILLISECOND, &nanos))
{
+ return Status::DataQualityError("Arrow timestamp {} overflows
nanoseconds", value);
+ }
+ break;
+ case arrow::TimeUnit::MICRO:
+ if (!checked_scale_to_nanos(value, NANOS_PER_MICROSECOND, &nanos))
{
+ return Status::DataQualityError("Arrow timestamp {} overflows
nanoseconds", value);
+ }
+ break;
+ case arrow::TimeUnit::NANO:
+ nanos = value;
+ break;
+ }
+ if (!type->timezone().empty()) {
+ RETURN_IF_ERROR(utc_epoch_nanos_to_local_epoch_nanos(nanos, ctz,
&nanos));
+ }
+ data.push_back(DateTimeV2NanoValue(nanos));
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::read_column_from_decoded_values(
+ IColumn& column, const DecodedColumnView& view) const {
+ if (view.value_kind != DecodedValueKind::INT64) {
+ return decoded_column_view_handle_conversion_failure(
+ column, view,
+ Status::NotSupported("DATETIMEV2 nano decoded reader expects
INT64 source"));
+ }
+ auto& data = assert_cast<ColumnDateTimeV2Nano&>(column).get_data();
+ const auto* values = reinterpret_cast<const int64_t*>(view.values);
+ static const auto utc = cctz::utc_time_zone();
+ const auto& timezone = view.timezone == nullptr ? utc : *view.timezone;
+ for (int64_t row = 0; row < view.row_count; ++row) {
+ if (decoded_column_view_row_is_null(view, row)) {
+ data.push_back(DateTimeV2NanoValue(0));
+ continue;
+ }
+ int64_t nanos = 0;
+ int64_t multiplier = 1;
+ switch (view.time_unit) {
+ case DecodedTimeUnit::MILLIS:
+ multiplier = NANOS_PER_MILLISECOND;
+ break;
+ case DecodedTimeUnit::MICROS:
+ multiplier = NANOS_PER_MICROSECOND;
+ break;
+ case DecodedTimeUnit::NANOS:
+ break;
+ case DecodedTimeUnit::UNKNOWN:
+ return decoded_column_view_handle_conversion_failure(
+ column, view,
+ Status::NotSupported("DATETIMEV2 nano decoded reader
requires a time unit"));
+ }
+ if (!checked_scale_to_nanos(values[row], multiplier, &nanos)) {
+ return decoded_column_view_handle_conversion_failure(
+ column, view,
+ Status::DataQualityError("Timestamp {} overflows
nanoseconds", values[row]));
+ }
+ if (view.timestamp_is_adjusted_to_utc) {
+ RETURN_IF_ERROR(utc_epoch_nanos_to_local_epoch_nanos(nanos,
timezone, &nanos));
+ }
+ data.push_back(DateTimeV2NanoValue(nanos));
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::write_column_to_mysql_binary(
+ const IColumn& column, MysqlRowBinaryBuffer& row_buffer, int64_t
row_idx, bool col_const,
+ const FormatOptions& options) const {
+ const auto& data = assert_cast<const
ColumnDateTimeV2Nano&>(column).get_data();
+ const auto index = index_check_const(row_idx, col_const);
+ const auto value = DateTimeV2NanoValue(data[index]).to_string(_scale);
+ if (row_buffer.push_string(value.data(), value.size()) != 0) {
Review Comment:
[P1] Keep MySQL binary metadata and value encoding consistent
This writes a length-encoded 29-byte ASCII string, but FE still advertises
the result as `MYSQL_TYPE_DATETIME`. Prepared-statement clients decode that
type using the temporal 0/4/7/11-byte layout (which the legacy SerDe emits), so
the leading `29` is not a valid DATETIME length and the row can be rejected or
corrupted. Since the standard temporal layout cannot carry nanoseconds, please
advertise a string-compatible type for this path or use another
protocol-compatible representation.
##########
fe/fe-core/src/test/java/org/apache/doris/http/ForwardToMasterTest.java:
##########
@@ -50,13 +50,14 @@ public void testAddBeDropBe() throws Exception {
JSONObject object = (JSONObject) JSONValue.parse(respStr);
JSONObject data = (JSONObject) object.get("data");
- JSONArray columnNames = (JSONArray) data.get("columnNames");
- JSONArray rows = (JSONArray) data.get("rows");
+ JSONArray columnNames = (JSONArray) ((JSONObject)
data.get("columnNames"))
Review Comment:
[P1] Keep the forwarded node response as JSON arrays
`NodeInfo` still serializes `columnNames` as `List<String>` and `rows` as
`List<List<String>>`, so JSON-simple returns a `JSONArray` for each field.
Casting either value to `JSONObject` throws before these assertions, and
flattening `rows` also contradicts the endpoint shape. Please restore direct
array extraction and row-by-row traversal (or remove this unrelated test
change).
##########
be/src/core/data_type_serde/data_type_datetimev2_nano_serde.cpp:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "core/data_type_serde/data_type_datetimev2_nano_serde.h"
+
+#include <arrow/array.h>
+#include <arrow/builder.h>
+#include <cctz/time_zone.h>
+
+#include <cctype>
+#include <limits>
+#include <orc/Vector.hh>
+#include <string>
+
+#include "common/config.h"
+#include "core/assert_cast.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/data_type_serde/arrow_validation.h"
+#include "core/data_type_serde/decoded_column_view.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/function/cast/cast_to_datetimev2_impl.hpp"
+#include "util/mysql_row_buffer.h"
+#include "util/unaligned.h"
+
+namespace doris {
+namespace {
+
+constexpr int64_t NANOS_PER_MILLISECOND = 1000000;
+constexpr int64_t NANOS_PER_MICROSECOND = 1000;
+
+bool checked_scale_to_nanos(int64_t value, int64_t multiplier, int64_t*
result) {
+ return !__builtin_mul_overflow(value, multiplier, result);
+}
+
+Status utc_epoch_nanos_to_local_epoch_nanos(int64_t source, const
cctz::time_zone& timezone,
+ int64_t* result) {
+ const DateTimeV2NanoValue source_value(source);
+ DateV2Value<DateTimeV2ValueType> local_value;
+ local_value.from_unixtime(source_value.epoch_seconds(), timezone);
+ local_value.set_microsecond(source_value.microsecond());
+ DateTimeV2NanoValue target_value;
+ if (!target_value.from_datetime(local_value,
source_value.nanosecond_remainder())) {
+ return Status::DataQualityError("Timestamp {} is outside DATETIMEV2
nanosecond range",
+ source);
+ }
+ *result = target_value.epoch_nanos();
+ return Status::OK();
+}
+
+Status local_epoch_nanos_to_utc_epoch_nanos(int64_t source, const
cctz::time_zone& timezone,
+ int64_t* result) {
+ const DateTimeV2NanoValue source_value(source);
+ auto local_value = source_value.to_datetime();
+ int64_t seconds = 0;
+ local_value.unix_timestamp(&seconds, timezone);
+ const __int128 epoch_nanos =
+ static_cast<__int128>(seconds) *
DateTimeV2NanoValue::NANOS_PER_SECOND +
+ source_value.nanosecond();
+ if (epoch_nanos < std::numeric_limits<int64_t>::min() ||
+ epoch_nanos > std::numeric_limits<int64_t>::max()) {
+ return Status::DataQualityError("DATETIMEV2 value {} is outside epoch
nanosecond range",
+ source_value.to_string(9));
+ }
+ *result = static_cast<int64_t>(epoch_nanos);
+ return Status::OK();
+}
+
+} // namespace
+
+Status parse_datetimev2_nano(StringRef str, int scale, int64_t* epoch_nanos,
+ const cctz::time_zone* local_time_zone) {
+ DORIS_CHECK_GE(scale, 7);
+ DORIS_CHECK_LE(scale, 9);
+
+ std::string input(str.data, str.size);
+ size_t fraction_begin = std::string::npos;
+ size_t fraction_end = std::string::npos;
+ const size_t dot = input.rfind('.');
+ if (dot != std::string::npos && dot + 1 < input.size() &&
+ std::isdigit(static_cast<unsigned char>(input[dot + 1]))) {
+ fraction_begin = dot + 1;
+ fraction_end = fraction_begin;
+ while (fraction_end < input.size() &&
+ std::isdigit(static_cast<unsigned char>(input[fraction_end]))) {
+ ++fraction_end;
+ }
+ }
+
+ uint32_t nanos = 0;
+ size_t fraction_length = 0;
+ if (fraction_begin != std::string::npos) {
+ fraction_length = fraction_end - fraction_begin;
+ const size_t copied_digits = std::min<size_t>(fraction_length, 9);
+ for (size_t i = 0; i < copied_digits; ++i) {
+ nanos = nanos * 10 + static_cast<uint32_t>(input[fraction_begin +
i] - '0');
+ }
+ for (size_t i = copied_digits; i < 9; ++i) {
+ nanos *= 10;
+ }
+ }
+
+ const auto quantum = static_cast<uint32_t>(int_exp10(9 - scale));
+ nanos = nanos / quantum * quantum;
+ if (fraction_length > static_cast<size_t>(scale) && input[fraction_begin +
scale] >= '5') {
+ nanos += quantum;
+ }
+
+ std::string base = input;
+ if (fraction_begin != std::string::npos) {
+ base.erase(dot, fraction_end - dot);
+ }
+ const StringRef base_ref(base.data(), base.size());
+ DateV2Value<DateTimeV2ValueType> datetime;
+ CastParameters params {.status = Status::OK(), .is_strict = true};
+ CastToDatetimeV2::from_string_strict_mode<DatelikeParseMode::STRICT>(
+ base_ref, datetime, local_time_zone, 0, params);
+ if (!params.status.ok()) {
+ return params.status;
+ }
+
+ if (nanos == DateTimeV2NanoValue::NANOS_PER_SECOND) {
+ if (!datetime.date_add_interval<TimeUnit::SECOND>(
+ TimeInterval {TimeUnit::SECOND, 1, false})) {
+ return Status::InvalidArgument("DATETIMEV2 value overflows while
rounding '{}'", input);
+ }
+ nanos = 0;
+ }
+ datetime.set_microsecond(nanos / NANOS_PER_MICROSECOND);
+ DateTimeV2NanoValue value;
+ if (!value.from_datetime(datetime, static_cast<uint16_t>(nanos %
NANOS_PER_MICROSECOND))) {
+ return Status::InvalidArgument(
+ "DATETIMEV2({}) value '{}' is outside [{}, {}]", scale, input,
+
DateTimeV2NanoValue(std::numeric_limits<int64_t>::min()).to_string(scale),
+
DateTimeV2NanoValue(std::numeric_limits<int64_t>::max()).to_string(scale));
+ }
+ *epoch_nanos = value.epoch_nanos();
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_batch(const ColumnString&
strings,
+ ColumnNullable& result,
+ const FormatOptions&
options) const {
+ auto& data =
assert_cast<ColumnDateTimeV2Nano&>(result.get_nested_column()).get_data();
+ auto& null_map = result.get_null_map_column().get_data();
+ result.resize(strings.size());
+ for (size_t i = 0; i < strings.size(); ++i) {
+ int64_t value = 0;
+ const auto status =
+ parse_datetimev2_nano(strings.get_data_at(i), _scale, &value,
options.timezone);
+ null_map[i] = !status.ok();
+ data[i] = DateTimeV2NanoValue(value);
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_strict_mode_batch(
+ const ColumnString& strings, IColumn& result, const FormatOptions&
options,
+ const NullMap::value_type* null_map) const {
+ auto& data = assert_cast<ColumnDateTimeV2Nano&>(result).get_data();
+ result.resize(strings.size());
+ for (size_t i = 0; i < strings.size(); ++i) {
+ if (null_map != nullptr && null_map[i]) {
+ continue;
+ }
+ int64_t value = 0;
+ RETURN_IF_ERROR(
+ parse_datetimev2_nano(strings.get_data_at(i), _scale, &value,
options.timezone));
+ data[i] = DateTimeV2NanoValue(value);
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string(StringRef& str, IColumn&
column,
+ const FormatOptions& options)
const {
+ int64_t value = 0;
+ RETURN_IF_ERROR(parse_datetimev2_nano(str, _scale, &value,
options.timezone));
+
assert_cast<ColumnDateTimeV2Nano&>(column).insert_value(DateTimeV2NanoValue(value));
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_strict_mode(StringRef& str,
IColumn& column,
+ const
FormatOptions& options) const {
+ return from_string(str, column, options);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::serialize_column_to_json(const IColumn&
column,
+ int64_t
start_idx, int64_t end_idx,
+ BufferWritable&
bw,
+ FormatOptions&
options) const {
+ SERIALIZE_COLUMN_TO_JSON();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::serialize_one_cell_to_json(const IColumn&
column,
+ int64_t
row_num, BufferWritable& bw,
+ FormatOptions&
options) const {
+ auto [column_ptr, index] = check_column_const_set_readability(column,
row_num);
+ if (_nesting_level > 1) {
+ bw.write('"');
+ }
+ const auto value =
+ assert_cast<const ColumnDateTimeV2Nano&,
TypeCheckOnRelease::DISABLE>(*column_ptr)
+ .get_element(index);
+ const std::string result = value.to_string(_scale);
+ bw.write(result.data(), result.size());
+ if (_nesting_level > 1) {
+ bw.write('"');
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::deserialize_column_from_json_vector(
+ IColumn& column, std::vector<Slice>& slices, uint64_t*
num_deserialized,
+ const FormatOptions& options) const {
+ DESERIALIZE_COLUMN_FROM_JSON_VECTOR();
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::deserialize_one_cell_from_json(
+ IColumn& column, Slice& slice, const FormatOptions& options) const {
+ if (_nesting_level > 1) {
+ slice.trim_quote();
+ }
+ StringRef str(slice.data, slice.size);
+ return from_string(str, column, options);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::write_column_to_arrow(const IColumn&
column,
Review Comment:
[P1] Register the nano primitive in Arrow schema conversion
Normal Arrow/Parquet output cannot reach this new SerDe:
`convert_to_arrow_type` handles only legacy `TYPE_DATETIMEV2`, so a scale-7..9
expression fails schema construction with `Unknown primitive
type(DateTimeV2(9))`. This also affects Arrow Flight/Python paths and nested
values. Please map the new primitive to an Arrow nanosecond timestamp and add
an end-to-end output test rather than only a manually constructed builder test.
##########
be/src/core/data_type/define_primitive_type.h:
##########
@@ -73,7 +73,9 @@ enum PrimitiveType : PrimitiveNative {
TYPE_UINT64, /* 39, used as offset */
TYPE_FIXED_LENGTH_OBJECT, /* 40, represent fixed-length object
on BE */
TYPE_VARBINARY, /* 41, varbinary */
- TYPE_TIMESTAMPTZ /* 42, timestamptz */
+ TYPE_TIMESTAMPTZ, /* 42, timestamptz */
+ // Internal physical type for DATETIMEV2(7..9). The SQL/Thrift type
remains DATETIMEV2.
+ TYPE_DATETIMEV2_NANO /* 43, signed Int64 epoch nanoseconds */
Review Comment:
[P1] Extend the JNI bridge for the new primitive
Scale-7..9 columns now carry `TYPE_DATETIMEV2_NANO`, but `JniDataBridge`
recognizes only legacy `TYPE_DATETIMEV2` in schema mapping, metadata, and
column transfer. JNI-backed external scans and Java UDF/UDAF/UDTF input/output
therefore hit unsupported branches (also for nested values). Because the raw
representation changed, this needs an explicit C++/Java wire contract rather
than an alias; please implement both directions and add a scale-9 round trip.
##########
be/src/core/data_type_serde/data_type_datetimev2_nano_serde.cpp:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "core/data_type_serde/data_type_datetimev2_nano_serde.h"
+
+#include <arrow/array.h>
+#include <arrow/builder.h>
+#include <cctz/time_zone.h>
+
+#include <cctype>
+#include <limits>
+#include <orc/Vector.hh>
+#include <string>
+
+#include "common/config.h"
+#include "core/assert_cast.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/data_type_serde/arrow_validation.h"
+#include "core/data_type_serde/decoded_column_view.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/function/cast/cast_to_datetimev2_impl.hpp"
+#include "util/mysql_row_buffer.h"
+#include "util/unaligned.h"
+
+namespace doris {
+namespace {
+
+constexpr int64_t NANOS_PER_MILLISECOND = 1000000;
+constexpr int64_t NANOS_PER_MICROSECOND = 1000;
+
+bool checked_scale_to_nanos(int64_t value, int64_t multiplier, int64_t*
result) {
+ return !__builtin_mul_overflow(value, multiplier, result);
+}
+
+Status utc_epoch_nanos_to_local_epoch_nanos(int64_t source, const
cctz::time_zone& timezone,
+ int64_t* result) {
+ const DateTimeV2NanoValue source_value(source);
+ DateV2Value<DateTimeV2ValueType> local_value;
+ local_value.from_unixtime(source_value.epoch_seconds(), timezone);
+ local_value.set_microsecond(source_value.microsecond());
+ DateTimeV2NanoValue target_value;
+ if (!target_value.from_datetime(local_value,
source_value.nanosecond_remainder())) {
+ return Status::DataQualityError("Timestamp {} is outside DATETIMEV2
nanosecond range",
+ source);
+ }
+ *result = target_value.epoch_nanos();
+ return Status::OK();
+}
+
+Status local_epoch_nanos_to_utc_epoch_nanos(int64_t source, const
cctz::time_zone& timezone,
+ int64_t* result) {
+ const DateTimeV2NanoValue source_value(source);
+ auto local_value = source_value.to_datetime();
+ int64_t seconds = 0;
+ local_value.unix_timestamp(&seconds, timezone);
+ const __int128 epoch_nanos =
+ static_cast<__int128>(seconds) *
DateTimeV2NanoValue::NANOS_PER_SECOND +
+ source_value.nanosecond();
+ if (epoch_nanos < std::numeric_limits<int64_t>::min() ||
+ epoch_nanos > std::numeric_limits<int64_t>::max()) {
+ return Status::DataQualityError("DATETIMEV2 value {} is outside epoch
nanosecond range",
+ source_value.to_string(9));
+ }
+ *result = static_cast<int64_t>(epoch_nanos);
+ return Status::OK();
+}
+
+} // namespace
+
+Status parse_datetimev2_nano(StringRef str, int scale, int64_t* epoch_nanos,
+ const cctz::time_zone* local_time_zone) {
+ DORIS_CHECK_GE(scale, 7);
+ DORIS_CHECK_LE(scale, 9);
+
+ std::string input(str.data, str.size);
+ size_t fraction_begin = std::string::npos;
+ size_t fraction_end = std::string::npos;
+ const size_t dot = input.rfind('.');
+ if (dot != std::string::npos && dot + 1 < input.size() &&
+ std::isdigit(static_cast<unsigned char>(input[dot + 1]))) {
+ fraction_begin = dot + 1;
+ fraction_end = fraction_begin;
+ while (fraction_end < input.size() &&
+ std::isdigit(static_cast<unsigned char>(input[fraction_end]))) {
+ ++fraction_end;
+ }
+ }
+
+ uint32_t nanos = 0;
+ size_t fraction_length = 0;
+ if (fraction_begin != std::string::npos) {
+ fraction_length = fraction_end - fraction_begin;
+ const size_t copied_digits = std::min<size_t>(fraction_length, 9);
+ for (size_t i = 0; i < copied_digits; ++i) {
+ nanos = nanos * 10 + static_cast<uint32_t>(input[fraction_begin +
i] - '0');
+ }
+ for (size_t i = copied_digits; i < 9; ++i) {
+ nanos *= 10;
+ }
+ }
+
+ const auto quantum = static_cast<uint32_t>(int_exp10(9 - scale));
+ nanos = nanos / quantum * quantum;
+ if (fraction_length > static_cast<size_t>(scale) && input[fraction_begin +
scale] >= '5') {
+ nanos += quantum;
+ }
+
+ std::string base = input;
+ if (fraction_begin != std::string::npos) {
+ base.erase(dot, fraction_end - dot);
+ }
+ const StringRef base_ref(base.data(), base.size());
+ DateV2Value<DateTimeV2ValueType> datetime;
+ CastParameters params {.status = Status::OK(), .is_strict = true};
+ CastToDatetimeV2::from_string_strict_mode<DatelikeParseMode::STRICT>(
+ base_ref, datetime, local_time_zone, 0, params);
+ if (!params.status.ok()) {
+ return params.status;
+ }
+
+ if (nanos == DateTimeV2NanoValue::NANOS_PER_SECOND) {
+ if (!datetime.date_add_interval<TimeUnit::SECOND>(
+ TimeInterval {TimeUnit::SECOND, 1, false})) {
+ return Status::InvalidArgument("DATETIMEV2 value overflows while
rounding '{}'", input);
+ }
+ nanos = 0;
+ }
+ datetime.set_microsecond(nanos / NANOS_PER_MICROSECOND);
+ DateTimeV2NanoValue value;
+ if (!value.from_datetime(datetime, static_cast<uint16_t>(nanos %
NANOS_PER_MICROSECOND))) {
+ return Status::InvalidArgument(
+ "DATETIMEV2({}) value '{}' is outside [{}, {}]", scale, input,
+
DateTimeV2NanoValue(std::numeric_limits<int64_t>::min()).to_string(scale),
+
DateTimeV2NanoValue(std::numeric_limits<int64_t>::max()).to_string(scale));
+ }
+ *epoch_nanos = value.epoch_nanos();
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_batch(const ColumnString&
strings,
+ ColumnNullable& result,
+ const FormatOptions&
options) const {
+ auto& data =
assert_cast<ColumnDateTimeV2Nano&>(result.get_nested_column()).get_data();
+ auto& null_map = result.get_null_map_column().get_data();
+ result.resize(strings.size());
+ for (size_t i = 0; i < strings.size(); ++i) {
+ int64_t value = 0;
+ const auto status =
+ parse_datetimev2_nano(strings.get_data_at(i), _scale, &value,
options.timezone);
+ null_map[i] = !status.ok();
+ data[i] = DateTimeV2NanoValue(value);
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_strict_mode_batch(
+ const ColumnString& strings, IColumn& result, const FormatOptions&
options,
+ const NullMap::value_type* null_map) const {
+ auto& data = assert_cast<ColumnDateTimeV2Nano&>(result).get_data();
+ result.resize(strings.size());
+ for (size_t i = 0; i < strings.size(); ++i) {
+ if (null_map != nullptr && null_map[i]) {
+ continue;
+ }
+ int64_t value = 0;
+ RETURN_IF_ERROR(
+ parse_datetimev2_nano(strings.get_data_at(i), _scale, &value,
options.timezone));
+ data[i] = DateTimeV2NanoValue(value);
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string(StringRef& str, IColumn&
column,
+ const FormatOptions& options)
const {
+ int64_t value = 0;
+ RETURN_IF_ERROR(parse_datetimev2_nano(str, _scale, &value,
options.timezone));
+
assert_cast<ColumnDateTimeV2Nano&>(column).insert_value(DateTimeV2NanoValue(value));
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_strict_mode(StringRef& str,
IColumn& column,
+ const
FormatOptions& options) const {
+ return from_string(str, column, options);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::serialize_column_to_json(const IColumn&
column,
+ int64_t
start_idx, int64_t end_idx,
+ BufferWritable&
bw,
+ FormatOptions&
options) const {
+ SERIALIZE_COLUMN_TO_JSON();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::serialize_one_cell_to_json(const IColumn&
column,
+ int64_t
row_num, BufferWritable& bw,
+ FormatOptions&
options) const {
+ auto [column_ptr, index] = check_column_const_set_readability(column,
row_num);
+ if (_nesting_level > 1) {
+ bw.write('"');
+ }
+ const auto value =
+ assert_cast<const ColumnDateTimeV2Nano&,
TypeCheckOnRelease::DISABLE>(*column_ptr)
+ .get_element(index);
+ const std::string result = value.to_string(_scale);
+ bw.write(result.data(), result.size());
+ if (_nesting_level > 1) {
+ bw.write('"');
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::deserialize_column_from_json_vector(
+ IColumn& column, std::vector<Slice>& slices, uint64_t*
num_deserialized,
+ const FormatOptions& options) const {
+ DESERIALIZE_COLUMN_FROM_JSON_VECTOR();
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::deserialize_one_cell_from_json(
+ IColumn& column, Slice& slice, const FormatOptions& options) const {
+ if (_nesting_level > 1) {
+ slice.trim_quote();
+ }
+ StringRef str(slice.data, slice.size);
+ return from_string(str, column, options);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::write_column_to_arrow(const IColumn&
column,
+ const NullMap*
null_map,
+ arrow::ArrayBuilder*
array_builder,
+ int64_t start,
int64_t end,
+ const
cctz::time_zone& ctz) const {
+ const auto& data = assert_cast<const
ColumnDateTimeV2Nano&>(column).get_data();
+ auto& builder = assert_cast<arrow::TimestampBuilder&>(*array_builder);
+ const auto timestamp_type =
std::static_pointer_cast<arrow::TimestampType>(builder.type());
+ const auto& timezone = timestamp_type->timezone();
+ for (int64_t i = start; i < end; ++i) {
+ if (null_map != nullptr && (*null_map)[i]) {
+ RETURN_IF_ERROR(checkArrowStatus(builder.AppendNull(), column,
builder));
+ continue;
+ }
+ int64_t value = data[i].epoch_nanos();
+ if (!timezone.empty()) {
+ RETURN_IF_ERROR(local_epoch_nanos_to_utc_epoch_nanos(value, ctz,
&value));
+ }
+ switch (timestamp_type->unit()) {
+ case arrow::TimeUnit::SECOND:
+ value /= DateTimeV2NanoValue::NANOS_PER_SECOND;
+ break;
+ case arrow::TimeUnit::MILLI:
+ value /= NANOS_PER_MILLISECOND;
+ break;
+ case arrow::TimeUnit::MICRO:
+ value /= NANOS_PER_MICROSECOND;
+ break;
+ case arrow::TimeUnit::NANO:
+ break;
+ }
+ RETURN_IF_ERROR(checkArrowStatus(builder.Append(value), column,
builder));
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::read_column_from_arrow(IColumn& column,
+ const arrow::Array*
arrow_array,
+ int64_t start,
int64_t end,
+ const
cctz::time_zone& ctz) const {
+ if (arrow_array->type_id() != arrow::Type::TIMESTAMP) {
+ return Status::InvalidArgument("Expected Arrow timestamp, got {}",
arrow_array->type_id());
+ }
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
+ const auto& array = assert_cast<const
arrow::TimestampArray&>(*arrow_array);
+ const auto type =
std::static_pointer_cast<arrow::TimestampType>(array.type());
+ auto& data = assert_cast<ColumnDateTimeV2Nano&>(column).get_data();
+ for (int64_t i = start; i < end; ++i) {
+ int64_t value = array.Value(i);
+ int64_t nanos = 0;
+ switch (type->unit()) {
+ case arrow::TimeUnit::SECOND:
+ if (!checked_scale_to_nanos(value,
DateTimeV2NanoValue::NANOS_PER_SECOND, &nanos)) {
+ return Status::DataQualityError("Arrow timestamp {} overflows
nanoseconds", value);
+ }
+ break;
+ case arrow::TimeUnit::MILLI:
+ if (!checked_scale_to_nanos(value, NANOS_PER_MILLISECOND, &nanos))
{
+ return Status::DataQualityError("Arrow timestamp {} overflows
nanoseconds", value);
+ }
+ break;
+ case arrow::TimeUnit::MICRO:
+ if (!checked_scale_to_nanos(value, NANOS_PER_MICROSECOND, &nanos))
{
+ return Status::DataQualityError("Arrow timestamp {} overflows
nanoseconds", value);
+ }
+ break;
+ case arrow::TimeUnit::NANO:
+ nanos = value;
+ break;
+ }
+ if (!type->timezone().empty()) {
+ RETURN_IF_ERROR(utc_epoch_nanos_to_local_epoch_nanos(nanos, ctz,
&nanos));
+ }
+ data.push_back(DateTimeV2NanoValue(nanos));
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::read_column_from_decoded_values(
+ IColumn& column, const DecodedColumnView& view) const {
+ if (view.value_kind != DecodedValueKind::INT64) {
+ return decoded_column_view_handle_conversion_failure(
+ column, view,
+ Status::NotSupported("DATETIMEV2 nano decoded reader expects
INT64 source"));
+ }
+ auto& data = assert_cast<ColumnDateTimeV2Nano&>(column).get_data();
+ const auto* values = reinterpret_cast<const int64_t*>(view.values);
+ static const auto utc = cctz::utc_time_zone();
+ const auto& timezone = view.timezone == nullptr ? utc : *view.timezone;
+ for (int64_t row = 0; row < view.row_count; ++row) {
+ if (decoded_column_view_row_is_null(view, row)) {
+ data.push_back(DateTimeV2NanoValue(0));
+ continue;
+ }
+ int64_t nanos = 0;
+ int64_t multiplier = 1;
+ switch (view.time_unit) {
+ case DecodedTimeUnit::MILLIS:
+ multiplier = NANOS_PER_MILLISECOND;
+ break;
+ case DecodedTimeUnit::MICROS:
+ multiplier = NANOS_PER_MICROSECOND;
+ break;
+ case DecodedTimeUnit::NANOS:
+ break;
+ case DecodedTimeUnit::UNKNOWN:
+ return decoded_column_view_handle_conversion_failure(
+ column, view,
+ Status::NotSupported("DATETIMEV2 nano decoded reader
requires a time unit"));
+ }
+ if (!checked_scale_to_nanos(values[row], multiplier, &nanos)) {
+ return decoded_column_view_handle_conversion_failure(
+ column, view,
+ Status::DataQualityError("Timestamp {} overflows
nanoseconds", values[row]));
+ }
+ if (view.timestamp_is_adjusted_to_utc) {
+ RETURN_IF_ERROR(utc_epoch_nanos_to_local_epoch_nanos(nanos,
timezone, &nanos));
+ }
+ data.push_back(DateTimeV2NanoValue(nanos));
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::write_column_to_mysql_binary(
+ const IColumn& column, MysqlRowBinaryBuffer& row_buffer, int64_t
row_idx, bool col_const,
+ const FormatOptions& options) const {
+ const auto& data = assert_cast<const
ColumnDateTimeV2Nano&>(column).get_data();
+ const auto index = index_check_const(row_idx, col_const);
+ const auto value = DateTimeV2NanoValue(data[index]).to_string(_scale);
+ if (row_buffer.push_string(value.data(), value.size()) != 0) {
+ return Status::InternalError("pack MySQL DATETIMEV2 nano buffer
failed");
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::write_column_to_orc(const std::string&
timezone,
Review Comment:
[P1] Register the nano primitive in ORC schema construction
The production ORC transformer builds its schema before this method is
called, and `_build_orc_type` recognizes only legacy `TYPE_DATETIMEV2`. A
scale-7..9 value therefore fails with `Unsupported type DateTimeV2(9) to build
orc type` (also recursively in arrays/maps/structs), even though the direct
SerDe unit test passes a prebuilt timestamp batch. Please add the new primitive
to ORC schema dispatch and cover end-to-end output.
##########
be/src/core/data_type/data_type_date_time.h:
##########
@@ -111,6 +111,8 @@ template <typename DataType>
constexpr bool IsDateTimeV2Type = false;
template <>
inline constexpr bool IsDateTimeV2Type<DataTypeDateTimeV2> = true;
+template <>
+inline constexpr bool IsDateTimeV2Type<DataTypeDateTimeV2Nano> = true;
Review Comment:
[P1] Do not reinterpret epoch nanoseconds as packed datetime
This specialization sends the new type through the existing numeric-cast
branches, which `reinterpret_cast` each raw value to the legacy packed
`DateTimeV2Value` layout before extracting date/time digits. The nano type
instead stores signed epoch nanoseconds, so stored-column casts to
BIGINT/FLOAT/DOUBLE silently return unrelated values (even epoch zero is not
packed 1970-01-01). Please add an explicit civil-time conversion for the nano
representation and cover non-folded numeric casts.
##########
be/src/exprs/create_predicate_function.h:
##########
@@ -92,6 +92,7 @@ class PredicateFunctionCreator {
M(TYPE_DATETIME) \
M(TYPE_DATEV2) \
M(TYPE_DATETIMEV2) \
+ M(TYPE_DATETIMEV2_NANO) \
Review Comment:
[P1] Add nano transport for global runtime filters
This enables local IN/min/max objects for `TYPE_DATETIMEV2_NANO`, but
distributed publication is incomplete. The producer instantiates
`get_convertor<DateTimeV2NanoValue>()`, which has no protobuf converter, and
the remote IN/min/max assignment switches also omit the new primitive. Thus an
equal-scale distributed join can fail even without any cast. Please add
matching raw-nanosecond serialization/deserialization and cover broadcast and
shuffle joins below the IN-to-bloom threshold.
##########
be/src/core/data_type_serde/data_type_datetimev2_nano_serde.cpp:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "core/data_type_serde/data_type_datetimev2_nano_serde.h"
+
+#include <arrow/array.h>
+#include <arrow/builder.h>
+#include <cctz/time_zone.h>
+
+#include <cctype>
+#include <limits>
+#include <orc/Vector.hh>
+#include <string>
+
+#include "common/config.h"
+#include "core/assert_cast.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/data_type_serde/arrow_validation.h"
+#include "core/data_type_serde/decoded_column_view.h"
+#include "core/value/vdatetime_value.h"
+#include "exprs/function/cast/cast_to_datetimev2_impl.hpp"
+#include "util/mysql_row_buffer.h"
+#include "util/unaligned.h"
+
+namespace doris {
+namespace {
+
+constexpr int64_t NANOS_PER_MILLISECOND = 1000000;
+constexpr int64_t NANOS_PER_MICROSECOND = 1000;
+
+bool checked_scale_to_nanos(int64_t value, int64_t multiplier, int64_t*
result) {
+ return !__builtin_mul_overflow(value, multiplier, result);
+}
+
+Status utc_epoch_nanos_to_local_epoch_nanos(int64_t source, const
cctz::time_zone& timezone,
+ int64_t* result) {
+ const DateTimeV2NanoValue source_value(source);
+ DateV2Value<DateTimeV2ValueType> local_value;
+ local_value.from_unixtime(source_value.epoch_seconds(), timezone);
+ local_value.set_microsecond(source_value.microsecond());
+ DateTimeV2NanoValue target_value;
+ if (!target_value.from_datetime(local_value,
source_value.nanosecond_remainder())) {
+ return Status::DataQualityError("Timestamp {} is outside DATETIMEV2
nanosecond range",
+ source);
+ }
+ *result = target_value.epoch_nanos();
+ return Status::OK();
+}
+
+Status local_epoch_nanos_to_utc_epoch_nanos(int64_t source, const
cctz::time_zone& timezone,
+ int64_t* result) {
+ const DateTimeV2NanoValue source_value(source);
+ auto local_value = source_value.to_datetime();
+ int64_t seconds = 0;
+ local_value.unix_timestamp(&seconds, timezone);
+ const __int128 epoch_nanos =
+ static_cast<__int128>(seconds) *
DateTimeV2NanoValue::NANOS_PER_SECOND +
+ source_value.nanosecond();
+ if (epoch_nanos < std::numeric_limits<int64_t>::min() ||
+ epoch_nanos > std::numeric_limits<int64_t>::max()) {
+ return Status::DataQualityError("DATETIMEV2 value {} is outside epoch
nanosecond range",
+ source_value.to_string(9));
+ }
+ *result = static_cast<int64_t>(epoch_nanos);
+ return Status::OK();
+}
+
+} // namespace
+
+Status parse_datetimev2_nano(StringRef str, int scale, int64_t* epoch_nanos,
+ const cctz::time_zone* local_time_zone) {
+ DORIS_CHECK_GE(scale, 7);
+ DORIS_CHECK_LE(scale, 9);
+
+ std::string input(str.data, str.size);
+ size_t fraction_begin = std::string::npos;
+ size_t fraction_end = std::string::npos;
+ const size_t dot = input.rfind('.');
+ if (dot != std::string::npos && dot + 1 < input.size() &&
+ std::isdigit(static_cast<unsigned char>(input[dot + 1]))) {
+ fraction_begin = dot + 1;
+ fraction_end = fraction_begin;
+ while (fraction_end < input.size() &&
+ std::isdigit(static_cast<unsigned char>(input[fraction_end]))) {
+ ++fraction_end;
+ }
+ }
+
+ uint32_t nanos = 0;
+ size_t fraction_length = 0;
+ if (fraction_begin != std::string::npos) {
+ fraction_length = fraction_end - fraction_begin;
+ const size_t copied_digits = std::min<size_t>(fraction_length, 9);
+ for (size_t i = 0; i < copied_digits; ++i) {
+ nanos = nanos * 10 + static_cast<uint32_t>(input[fraction_begin +
i] - '0');
+ }
+ for (size_t i = copied_digits; i < 9; ++i) {
+ nanos *= 10;
+ }
+ }
+
+ const auto quantum = static_cast<uint32_t>(int_exp10(9 - scale));
+ nanos = nanos / quantum * quantum;
+ if (fraction_length > static_cast<size_t>(scale) && input[fraction_begin +
scale] >= '5') {
+ nanos += quantum;
+ }
+
+ std::string base = input;
+ if (fraction_begin != std::string::npos) {
+ base.erase(dot, fraction_end - dot);
+ }
+ const StringRef base_ref(base.data(), base.size());
+ DateV2Value<DateTimeV2ValueType> datetime;
+ CastParameters params {.status = Status::OK(), .is_strict = true};
+ CastToDatetimeV2::from_string_strict_mode<DatelikeParseMode::STRICT>(
+ base_ref, datetime, local_time_zone, 0, params);
+ if (!params.status.ok()) {
+ return params.status;
+ }
+
+ if (nanos == DateTimeV2NanoValue::NANOS_PER_SECOND) {
+ if (!datetime.date_add_interval<TimeUnit::SECOND>(
+ TimeInterval {TimeUnit::SECOND, 1, false})) {
+ return Status::InvalidArgument("DATETIMEV2 value overflows while
rounding '{}'", input);
+ }
+ nanos = 0;
+ }
+ datetime.set_microsecond(nanos / NANOS_PER_MICROSECOND);
+ DateTimeV2NanoValue value;
+ if (!value.from_datetime(datetime, static_cast<uint16_t>(nanos %
NANOS_PER_MICROSECOND))) {
+ return Status::InvalidArgument(
+ "DATETIMEV2({}) value '{}' is outside [{}, {}]", scale, input,
+
DateTimeV2NanoValue(std::numeric_limits<int64_t>::min()).to_string(scale),
+
DateTimeV2NanoValue(std::numeric_limits<int64_t>::max()).to_string(scale));
+ }
+ *epoch_nanos = value.epoch_nanos();
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_batch(const ColumnString&
strings,
+ ColumnNullable& result,
+ const FormatOptions&
options) const {
+ auto& data =
assert_cast<ColumnDateTimeV2Nano&>(result.get_nested_column()).get_data();
+ auto& null_map = result.get_null_map_column().get_data();
+ result.resize(strings.size());
+ for (size_t i = 0; i < strings.size(); ++i) {
+ int64_t value = 0;
+ const auto status =
+ parse_datetimev2_nano(strings.get_data_at(i), _scale, &value,
options.timezone);
+ null_map[i] = !status.ok();
+ data[i] = DateTimeV2NanoValue(value);
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_strict_mode_batch(
+ const ColumnString& strings, IColumn& result, const FormatOptions&
options,
+ const NullMap::value_type* null_map) const {
+ auto& data = assert_cast<ColumnDateTimeV2Nano&>(result).get_data();
+ result.resize(strings.size());
+ for (size_t i = 0; i < strings.size(); ++i) {
+ if (null_map != nullptr && null_map[i]) {
+ continue;
+ }
+ int64_t value = 0;
+ RETURN_IF_ERROR(
+ parse_datetimev2_nano(strings.get_data_at(i), _scale, &value,
options.timezone));
+ data[i] = DateTimeV2NanoValue(value);
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string(StringRef& str, IColumn&
column,
+ const FormatOptions& options)
const {
+ int64_t value = 0;
+ RETURN_IF_ERROR(parse_datetimev2_nano(str, _scale, &value,
options.timezone));
+
assert_cast<ColumnDateTimeV2Nano&>(column).insert_value(DateTimeV2NanoValue(value));
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::from_string_strict_mode(StringRef& str,
IColumn& column,
+ const
FormatOptions& options) const {
+ return from_string(str, column, options);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::serialize_column_to_json(const IColumn&
column,
+ int64_t
start_idx, int64_t end_idx,
+ BufferWritable&
bw,
+ FormatOptions&
options) const {
+ SERIALIZE_COLUMN_TO_JSON();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::serialize_one_cell_to_json(const IColumn&
column,
+ int64_t
row_num, BufferWritable& bw,
+ FormatOptions&
options) const {
+ auto [column_ptr, index] = check_column_const_set_readability(column,
row_num);
+ if (_nesting_level > 1) {
+ bw.write('"');
+ }
+ const auto value =
+ assert_cast<const ColumnDateTimeV2Nano&,
TypeCheckOnRelease::DISABLE>(*column_ptr)
+ .get_element(index);
+ const std::string result = value.to_string(_scale);
+ bw.write(result.data(), result.size());
+ if (_nesting_level > 1) {
+ bw.write('"');
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::deserialize_column_from_json_vector(
+ IColumn& column, std::vector<Slice>& slices, uint64_t*
num_deserialized,
+ const FormatOptions& options) const {
+ DESERIALIZE_COLUMN_FROM_JSON_VECTOR();
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::deserialize_one_cell_from_json(
+ IColumn& column, Slice& slice, const FormatOptions& options) const {
+ if (_nesting_level > 1) {
+ slice.trim_quote();
+ }
+ StringRef str(slice.data, slice.size);
+ return from_string(str, column, options);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::write_column_to_arrow(const IColumn&
column,
+ const NullMap*
null_map,
+ arrow::ArrayBuilder*
array_builder,
+ int64_t start,
int64_t end,
+ const
cctz::time_zone& ctz) const {
+ const auto& data = assert_cast<const
ColumnDateTimeV2Nano&>(column).get_data();
+ auto& builder = assert_cast<arrow::TimestampBuilder&>(*array_builder);
+ const auto timestamp_type =
std::static_pointer_cast<arrow::TimestampType>(builder.type());
+ const auto& timezone = timestamp_type->timezone();
+ for (int64_t i = start; i < end; ++i) {
+ if (null_map != nullptr && (*null_map)[i]) {
+ RETURN_IF_ERROR(checkArrowStatus(builder.AppendNull(), column,
builder));
+ continue;
+ }
+ int64_t value = data[i].epoch_nanos();
+ if (!timezone.empty()) {
+ RETURN_IF_ERROR(local_epoch_nanos_to_utc_epoch_nanos(value, ctz,
&value));
+ }
+ switch (timestamp_type->unit()) {
+ case arrow::TimeUnit::SECOND:
+ value /= DateTimeV2NanoValue::NANOS_PER_SECOND;
+ break;
+ case arrow::TimeUnit::MILLI:
+ value /= NANOS_PER_MILLISECOND;
+ break;
+ case arrow::TimeUnit::MICRO:
+ value /= NANOS_PER_MICROSECOND;
+ break;
+ case arrow::TimeUnit::NANO:
+ break;
+ }
+ RETURN_IF_ERROR(checkArrowStatus(builder.Append(value), column,
builder));
+ }
+ return Status::OK();
+}
+
+Status DataTypeDateTimeV2NanoSerDe::read_column_from_arrow(IColumn& column,
+ const arrow::Array*
arrow_array,
+ int64_t start,
int64_t end,
+ const
cctz::time_zone& ctz) const {
+ if (arrow_array->type_id() != arrow::Type::TIMESTAMP) {
+ return Status::InvalidArgument("Expected Arrow timestamp, got {}",
arrow_array->type_id());
+ }
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
+ const auto& array = assert_cast<const
arrow::TimestampArray&>(*arrow_array);
+ const auto type =
std::static_pointer_cast<arrow::TimestampType>(array.type());
+ auto& data = assert_cast<ColumnDateTimeV2Nano&>(column).get_data();
+ for (int64_t i = start; i < end; ++i) {
+ int64_t value = array.Value(i);
+ int64_t nanos = 0;
+ switch (type->unit()) {
+ case arrow::TimeUnit::SECOND:
+ if (!checked_scale_to_nanos(value,
DateTimeV2NanoValue::NANOS_PER_SECOND, &nanos)) {
+ return Status::DataQualityError("Arrow timestamp {} overflows
nanoseconds", value);
+ }
+ break;
+ case arrow::TimeUnit::MILLI:
+ if (!checked_scale_to_nanos(value, NANOS_PER_MILLISECOND, &nanos))
{
+ return Status::DataQualityError("Arrow timestamp {} overflows
nanoseconds", value);
+ }
+ break;
+ case arrow::TimeUnit::MICRO:
+ if (!checked_scale_to_nanos(value, NANOS_PER_MICROSECOND, &nanos))
{
+ return Status::DataQualityError("Arrow timestamp {} overflows
nanoseconds", value);
+ }
+ break;
+ case arrow::TimeUnit::NANO:
+ nanos = value;
+ break;
+ }
+ if (!type->timezone().empty()) {
+ RETURN_IF_ERROR(utc_epoch_nanos_to_local_epoch_nanos(nanos, ctz,
&nanos));
+ }
+ data.push_back(DateTimeV2NanoValue(nanos));
Review Comment:
[P1] Normalize imported values to the declared scale
Both external readers append raw nanoseconds without consulting `_scale`. A
NANO value ending in `.123456789` can therefore retain all nine digits in a
`DATETIMEV2(7)` column, even though it prints as `.1234567`; equality, hashing,
and ordering still see the hidden digits. String ingestion of the same value
rounds to `.1234568`, so ingestion paths disagree. Please round/carry to the
destination scale before insertion and cover scale-7/8 external values.
##########
be/src/core/data_type_serde/data_type_serde.cpp:
##########
@@ -847,7 +884,20 @@ Status
DataTypeDateTimeV2SerDe::read_column_from_orc(IColumn& column,
if (orc_decode_row_count(view.rows, view.selected_rows) == 0) {
return Status::OK();
}
- return decode_timestamp_orc_values(column, view, *view.timezone);
+ return decode_datetimev2_orc_values(column, view, *view.timezone);
+}
+
+Status DataTypeDateTimeV2NanoSerDe::read_column_from_orc(IColumn& column,
Review Comment:
[P1] Route nanosecond file timestamps to the nano reader
The production schema mappers still infer legacy `DATETIMEV2(6)` for Parquet
`TIMESTAMP(NANOS)` and ORC `TIMESTAMP`. `_decode_column` consequently selects
the legacy SerDe (the ORC path divides nanoseconds by 1000), so this new nano
dispatch is bypassed and sub-microsecond input is lost. Please preserve
nanosecond precision in both reader generations and add an end-to-end file scan
instead of only direct SerDe tests.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java:
##########
@@ -254,14 +254,13 @@ static Result<String, AnalysisException> normalize(String
s) {
sb.append(":00");
}
- // parse MicroSecond
- // Keep up to 7 digits at most, 7th digit is use for overflow.
+ // Parse fractional seconds. Java's NANO_OF_SECOND supports up to 9
digits.
int j = i;
if (partNumber == 6 && i < s.length() && s.charAt(i) == '.') {
sb.append(s.charAt(i));
i += 1;
while (i < s.length() && Character.isDigit(s.charAt(i))) {
- if (i - j <= 7) {
+ if (i - j <= DateTimeV2Type.MAX_SCALE) {
Review Comment:
[P1] Preserve the guard digit for scale-9 rounding
This now keeps only nine fractional digits, but scale-9 rounding needs the
tenth digit. For example, FE folding truncates `.9999999995` to `.999999999`,
while the BE parser checks the next digit and carries it to the following
second. The old microsecond code intentionally kept a seventh guard digit;
please retain the equivalent tenth digit (or reject excess precision
consistently) and add folded/unfolded boundary cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
