morningman commented on code in PR #69016:
URL: https://github.com/apache/doris/pull/69016#discussion_r4259537450
##########
be/src/format_v2/parquet/parquet_reader.cpp:
##########
@@ -74,104 +74,149 @@ const ParquetColumnSchema* schema_child_by_name(const
ParquetColumnSchema& schem
return child_it == schema.children.end() ? nullptr : child_it->get();
}
-bool collect_variant_residual_leaf_ids(const ParquetColumnSchema& schema,
- const format::LocalColumnIndex&
projection,
- std::vector<int>* residual_leaf_ids) {
+bool collect_variant_terminal_fallback_leaf_ids(const ParquetColumnSchema&
schema,
+ const
format::LocalColumnIndex& projection,
+ std::vector<int>*
residual_leaf_ids) {
DORIS_CHECK(residual_leaf_ids != nullptr);
const auto* value = schema_child_by_name(schema, "value");
const auto* typed_value = schema_child_by_name(schema, "typed_value");
- if (value != nullptr && typed_value != nullptr) {
- if (value->kind != ParquetColumnSchemaKind::PRIMITIVE ||
value->leaf_column_id < 0) {
+ if (value != nullptr && value->kind == ParquetColumnSchemaKind::PRIMITIVE
&&
+ typed_value != nullptr) {
+ if (value->leaf_column_id < 0) {
return false;
}
+ const auto typed_projection_it =
+ std::ranges::find_if(projection.children, [&](const auto&
child_projection) {
+ return child_projection.local_id() ==
typed_value->local_id;
+ });
+ if (typed_projection_it == projection.children.end()) {
+ return false;
+ }
+ if (typed_value->kind != ParquetColumnSchemaKind::PRIMITIVE) {
+ return collect_variant_terminal_fallback_leaf_ids(*typed_value,
*typed_projection_it,
Review Comment:
Not changed here. The pairing this needs, a BE with this change behind an FE
that sends `[v, k]` for a generator argument, does not come out of a release.
#66575 shipped this BE rule in the same commit as the FE side that keeps the
whole container of a generator's Variant argument (the "A generator may change
the Variant container kind" branch in `AccessPathPlanCollector`), first in
4.1.4. branch-4.1 and branch-4.2 FEs both have it. An older FE cannot plan
these reads at all: 4.1.3 maps Iceberg VARIANT to UNSUPPORTED and has no Paimon
VARIANT read mapping. So the only FE that sends `[v, k]` here is master before
this PR. A BE-first upgrade from a released FE (4.1.4 or later, or 4.2) keeps
the container paths, and this PR brings master to the FE/BE pairing that 4.1.4
shipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
