github-actions[bot] commented on code in PR #66334:
URL: https://github.com/apache/doris/pull/66334#discussion_r3689724662
##########
be/src/format/jni/jni_data_bridge.cpp:
##########
@@ -177,6 +181,27 @@ Status
JniDataBridge::_fill_varbinary_column(TableMetaAddress& address,
return Status::OK();
}
+Status JniDataBridge::_fill_variant_v2_column(TableMetaAddress& address,
+ MutableColumnPtr& doris_column,
size_t num_rows) {
+ // VectorColumnVariant publishes these EncodedDataView fields immediately
after the null map.
+ const auto metadata_count =
static_cast<size_t>(address.next_meta_as_long());
+ const auto* metadata_offsets = reinterpret_cast<const
uint32_t*>(address.next_meta_as_ptr());
+ const auto* metadata_bytes = reinterpret_cast<const
char*>(address.next_meta_as_ptr());
+ const auto* metadata_ids = reinterpret_cast<const
uint32_t*>(address.next_meta_as_ptr());
+ const auto* value_offsets = reinterpret_cast<const
uint32_t*>(address.next_meta_as_ptr());
+ const auto* value_bytes = reinterpret_cast<const
char*>(address.next_meta_as_ptr());
+
+ auto& variant_column = assert_cast<ColumnVariantV2&>(*doris_column);
Review Comment:
[P2] Gate this ABI on the Variant V2 physical type. TYPE_VARIANT also backs
the legacy DataTypeVariant/ColumnVariant. During a BE-first rolling upgrade an
old FE sends Paimon Variant without variant_is_v2, so DataTypeFactory builds
ColumnVariant; the changed bridge now advertises the Java variant ABI and
reaches this unconditional ColumnVariantV2 cast, which throws instead of
rejecting the incompatible representation. Check the concrete data type/column
before advertising and filling this ABI (or explicitly convert it), and cover
the legacy DataTypeVariant case.
##########
be/src/format/jni/jni_data_bridge.cpp:
##########
@@ -146,6 +147,9 @@ Status JniDataBridge::fill_column(TableMetaAddress&
address, ColumnPtr& doris_co
case PrimitiveType::TYPE_VARBINARY:
status = _fill_varbinary_column(address, data_column, num_rows);
break;
+ case PrimitiveType::TYPE_VARIANT:
Review Comment:
[P1] Cover or reject the legacy scanner route for Variant. SET
enable_file_scanner_v2=false is still a supported, forwarded option and sends a
forced Paimon JNI split through legacy FileScanner/PaimonJniReader, whose
separate JniConnector maps Variant to unsupported and has no Variant fill path.
Non-null values throw; for a SQL NULL, unsupported reserve leaves nullMap at
zero and appendNull writes through address 0. Share/mirror the Variant bridge
support, or force/reject this scanner choice before Java allocation, and cover
the false-setting route.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java:
##########
@@ -342,6 +343,8 @@ private static Type
paimonPrimitiveTypeToDorisType(org.apache.paimon.types.DataT
return ScalarType.createTimeStampTzType(tsScale);
}
return ScalarType.createDatetimeV2Type(tsScale);
+ case VARIANT:
+ return VariantType.COMPUTE_V2_INSTANCE;
Review Comment:
[P1] Keep Paimon Variant projections on the JNI route. This maps the column
to a compute-v2 Variant, but with the default force_jni_scanner=false,
raw-convertible Parquet/ORC splits are still selected by supportNativeReader
based only on their suffix. Paimon's physical Variant is a value/metadata
group; the native Parquet schema path treats that group as Struct and then
rejects this Variant destination, so an ordinary SELECT fails while the new
regression passes only because it forces JNI. Gate native split selection when
any projected Paimon type contains Variant (recursively), and add
default-setting coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
