github-actions[bot] commented on code in PR #69382:
URL: https://github.com/apache/doris/pull/69382#discussion_r4334582627
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java:
##########
@@ -275,7 +278,8 @@ static Map<String, String>
getIdentityPartitionInfoMap(PartitionData partitionDa
continue;
}
- String columnName =
table.schema().findColumnName(partitionField.sourceId());
+ String columnName =
+ topLevelSourceColumnName(scanSchema,
partitionField.sourceId());
Review Comment:
[P2] Keep identity source lookup indexed on the per-file path
`computePerFileInvariants` calls this method for every distinct data file on
eager and streaming scans. `topLevelSourceColumnName` scans all of
`scanSchema.columns()` for each identity field, replacing the cached map lookup
in `Schema.findColumnName`. For a 1,000-column table whose identity partition
source is last, planning 1,000,000 files now performs about one billion
field-ID comparisons even for a narrow projection; the scratch cache only saves
repeated byte slices of the same file. Use the cached top-level lookup
[`scanSchema.asStruct().field(sourceId)`](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/api/src/main/java/org/apache/iceberg/types/Types.java#L987-L990),
or a scan-scoped ID-to-name map, to preserve pinned names and nested-source
exclusion without multiplying planning work by schema width.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
