github-actions[bot] commented on code in PR #69581:
URL: https://github.com/apache/doris/pull/69581#discussion_r4390935904
##########
be/src/core/data_type_serde/data_type_datev2_serde.cpp:
##########
@@ -277,11 +277,27 @@ Status
DataTypeDateV2SerDe::read_column_from_arrow(IColumn& column, const arrow:
const auto* base_ptr = reinterpret_cast<const
uint8_t*>(concrete_array->raw_values());
const size_t element_size = sizeof(int32_t);
for (auto value_i = start; value_i < end; ++value_i) {
+ // A null slot has no calendar value and its payload may be
outside the DATE range.
+ if (concrete_array->IsNull(value_i)) {
+ col_data.emplace_back(DateV2Value<DateV2ValueType>());
+ continue;
+ }
const uint8_t* raw_byte_ptr = base_ptr + value_i * element_size;
auto date_value = unaligned_load<int32_t>(raw_byte_ptr);
+ // Arrow allows any int32 day, Doris DATE only day numbers
1..DATE_MAX_DAYNR
+ // (0000-01-01..9999-12-31). For example, 2932897 is 10000-01-01:
its day number
+ // 3652425 makes get_date_from_daynr() fail, and ignoring that
result used to append
+ // a non-null 0000-00-00. Add in 64 bits so INT32_MAX cannot
overflow, and reject
+ // the value like the DATE64 branch above.
+ const int64_t daynr = static_cast<int64_t>(date_value) +
date_threshold;
DateV2Value<DateV2ValueType> v;
- v.get_date_from_daynr(date_value + date_threshold);
+ if (daynr <= 0 || daynr > DATE_MAX_DAYNR ||
Review Comment:
[P2] Ignore DATE32 payloads hidden by null fixed-size lists
A valid Lance v2.1 `FIXED_SIZE_LIST<DATE32,1>` using
`lance-encoding:structural-encoding=miniblock` can have parent validity
`[true,false,true]` and non-null child days `[0,2932897,1]`. Its visible result
is `[1970-01-01], NULL, [1970-01-02]`. The pinned [writer retains the child
data](https://github.com/lance-format/lance/blob/68c12dfd7efe02f90ad2d7f3a239a7eb64884e57/rust/lance-encoding/src/data.rs#L1995-L2004),
and the [decoder restores only outer
validity](https://github.com/lance-format/lance/blob/68c12dfd7efe02f90ad2d7f3a239a7eb64884e57/rust/lance-encoding/src/encodings/logical/primitive.rs#L5141-L5170).
This compact array passes normalization unchanged; NullableSerDe records the
parent NULL, but ArraySerDe still decodes all three children. The middle child
itself is non-null, so this new check rejects its hidden `10000-01-01` payload
and fails the scan. Propagate parent nulls into child conversion or skip hidden
spans, while retaining rejection for visible invalid dates; add
a native mixed-null fixed-list regression. This is distinct from resolved
comment 4369909226, which concerned visible out-of-range dates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
