924060929 opened a new pull request, #66473:
URL: https://github.com/apache/doris/pull/66473
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Queries with repeated equivalent external-table relations, such as UNION
branches over the same table snapshot, independently plan the same remote files
for every branch. On metadata-heavy Iceberg tables this repeats manifest
planning and can add seconds to FE planning time; Paimon, Hudi, and Hive have
the same class of duplicated split-planning work.
This change introduces a statement-execution-scoped, single-flight external
scan-task cache. Each connector builds a semantic key from the facts that
affect its split set:
- Iceberg: catalog/table identity, snapshot, schema, predicate, case
sensitivity, and task type.
- Paimon: relation/target identity, snapshot/schema, effective options,
projection, and predicate.
- Hudi: query/incremental instant, reader mode, partition identity,
runtime-prune mode, and Hudi parameters.
- Hive: catalog/table identity and immutable partition/file-listing inputs.
Equivalent scans reuse connector-native tasks within one statement
execution. Iceberg streaming/lazy batch paths and transactional Hive paths
remain uncached to preserve their memory and transaction semantics. Mutable
Hudi/Hive scheduling objects are copied before use so one scan cannot mutate
cached state observed by another scan.
The cache is represented by an execution-generation handle captured by each
ScanNode. PreparedStatement execution reset swaps in a new generation and
invalidates the old one, so delayed workers cannot repopulate or block the next
execution. Statement close and binary prepared-execution completion invalidate
and clear retained task lists, preventing idle PreparedStatements from
retaining split metadata. Loader failures are propagated to all waiters and are
not cached, allowing retry.
### Release note
Reuse equivalent Iceberg, Paimon, Hudi, and Hive scan-planning results
within one statement execution to reduce repeated external metadata work.
### Check List (For Author)
- Test: Unit Test
- `IcebergScanNodeTest`
- `PaimonScanNodeTest`
- `HudiScanNodeTest`
- `HiveScanNodeTest`
- `StatementContextTest`
- 103 tests passed on the latest `branch-4.1` base
- FE reactor build and Checkstyle passed
- Behavior changed: Yes. Equivalent external scans reuse statement-scoped
planning results; transactional Hive and streaming Iceberg paths retain their
previous behavior.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
