924060929 opened a new pull request, #69299:
URL: https://github.com/apache/doris/pull/69299
### What problem does this PR solve?
Issue Number: None
Problem Summary: Ordinary Iceberg scans and SQL result-cache validation
materialize and copy the full Doris partition view even when native file
pruning or a pinned metadata version is sufficient. Cache immutable engine
partition views and their pruning index by connector metadata generation, defer
conversion for native planning, and obtain SQL cache versions through a
source-neutral version provider. Record distinct current, historical and
branch/tag references when a statement reads several versions of one table, and
revalidate them through the existing snapshot resolver. Reuse
statement-retained table handles and combine relation-name discovery through
the connector SPI to avoid repeated remote existence and listing requests.
Preserve MV/partition metadata consumers, refresh invalidation and
frozen-reader schema consistency.
### Release note
Iceberg ordinary scans use native file pruning by default. The session
variable enable_iceberg_internal_partition_pruning enables Doris internal
partition pruning when needed. SQL result caching supports independently
versioned Iceberg snapshot and branch/tag references without enumerating
partitions.
### Check List (For Author)
- Test: Unit Test
- run-fe-ut.sh: 325 targeted metadata, MVCC, partition view, SQL cache
and dependent test cases passed
- run-fe-ut.sh: all 85 connector SPI tests passed separately
- build.sh --fe -j4 passed, including Checkstyle and connector
architecture gates
- Behavior changed: Yes, defer ordinary Iceberg partition conversion by
default and support version-aware external SQL caching
- Does this need documentation: Yes, updated fe/fe-connector/README.md
### Performance validation
Measured independently against unoptimized master `e0bf4874070`, with
identical Linux AArch64 Release BE, data/footer prewarming, and session
settings. Numbers are complete SQL medians, in milliseconds. The baseline
column shows two separate baseline runs.
| Scenario | Baseline runs | This PR |
|---|---:|---:|
| 120,000 partitions, warm LIMIT 10 | 389.38–463.29 | 9.69 |
| 120,000 partitions, warm EXPLAIN | 188.19–189.07 | 3.45 |
| 120,000 partitions, warm LIMIT 10 with internal pruning enabled |
333.18–391.80 | 9.76 |
| 1,000 manifests / 10,000 files, cold LIMIT 10 | 925.44–933.77 | 26.50 |
The large-partition protocol enables SQL Cache validation but varies an
always-true predicate to avoid result-cache hits. Ordinary LIMIT cases have 40
measured samples; EXPLAIN has 20. Baseline EXPLAIN retention is bounded by
using equivalent SELECT warmup, identically for all variants. Function-heavy
partition predicates remain approximately 5.5 seconds in this fixture and are
not claimed as a benefit.
### Additional validation
- Live SQL Cache checks: 170 assertions for current reads, snapshots,
timestamps, branch/tag movement, multiple versions of one table, schema
evolution/rename, snapshot expiry and ref removal/recreation.
- The final combined tree passed a clean FE build, 997 scoped unit tests
without skips, and the Hive streaming / Iceberg scheduled-split /
table-handle-reuse regression suites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
