morningman commented on code in PR #69547:
URL: https://github.com/apache/doris/pull/69547#discussion_r4366910227
##########
be/src/format/table/iceberg_reader.cpp:
##########
@@ -570,6 +668,44 @@ Status
IcebergParquetReader::on_before_init_reader(ReaderInitContext* ctx) {
}
return nullptr;
};
+ std::unordered_map<int32_t, const SlotDescriptor*> projected_slots;
+ for (const auto* slot : ctx->tuple_descriptor->slots()) {
+ projected_slots.emplace(slot->col_unique_id(), slot);
+ }
+ // Finds a nested key inside its root struct when the query projects that
root, so the key is
+ // taken from the projected column instead of reading the root a second
time.
+ const auto find_projected_root = [&](int32_t field_id, const DataTypePtr&
key_type,
+ std::string* root_name,
+ std::vector<size_t>* child_indexes)
-> Status {
+ const auto key_path = _find_schema_field_path(field_id);
+ if (key_path.empty()) {
+ return Status::OK();
+ }
+ DORIS_CHECK(key_path.front()->__isset.id);
+ const auto slot = projected_slots.find(key_path.front()->id);
+ if (slot == projected_slots.end()) {
+ return Status::OK();
+ }
+ std::vector<const schema::external::TField*> root_path;
+ if (!_find_schema_field_path_in_root(_current_schema_root(),
key_path.front()->id,
+ &root_path)) {
+ return Status::OK();
+ }
+ DORIS_CHECK(root_path.size() == 1);
+ DataTypePtr leaf_type;
+ bool found = false;
+ RETURN_IF_ERROR(find_projected_equality_key_path(*root_path.front(),
key_path,
+ slot->second->type(),
child_indexes,
+ &leaf_type, &found));
+ // The projected leaf has the table's current type. A key written
before a type promotion
+ // (INT promoted to BIGINT) keeps its own type, so it still reads the
physical root.
+ if (found &&
remove_nullable(leaf_type)->equals(*remove_nullable(key_type))) {
Review Comment:
Confirmed, and fixed in `9858972766f`. With a plan from an FE that predates
`iceberg_scan_semantics_version`, the projected struct is mapped level by
level, and a level whose file children do not all carry field IDs is matched by
name (kept for old FE plans by #65784), while the hidden key is located by
field ID. `find_projected_root` checked only the key's logical path and type,
so for a file `payload{existing without an ID, k#3}` with `k` renamed it took
the key from the projected `renamed_k`, which that mapping reads as NULL.
Before taking a key from the projected root, the reader now checks that the
projected column's schema mapping resolves the root and every level of the
key's path to the same file fields as the key's physical path, the way the
struct reader resolves each child (`children_file_column_name`); when it does
not, the key reads its own carrier, which the multimap change already supports.
`v1_parquet_legacy_plan_key_rereads_root_when_projected_child_matches_by_nam
e` covers this file under a legacy plan (without the check, id 2 survives the
delete) and under a plan with the marker, where both sides bind `k#3` by field
ID and the key still comes from the projected payload.
On reachability: plans that carry the marker resolve both sides by field ID,
and current FEs also send the Parquet timestamp marker, which routes Iceberg
scans to FileScannerV2. So this case needs an FE at 4.1.3 or older during a
BE-first upgrade, plus a data file whose nested struct mixes ID-tagged and
ID-less children, which Iceberg writers, Doris's included, do not produce.
branch-4.1 (since 4.1.4) and branch-4.2 resolve the projected root the same
way; we will report it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
