924060929 opened a new pull request, #70410: URL: https://github.com/apache/doris/pull/70410
### Background Iceberg batch scans either use a bounded completion-ordered manifest loader with the parsed Manifest Cache enabled, or SDK file planning with the cache disabled. SDK parallel file planning polls an empty output queue with a fixed 10 ms sleep. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Performance Problem Summary: Use the existing bounded manifest loader for supported cache-disabled data scans too. Parsed-result retention only selects lookup/load functions, while admission, completion notification, cancellation and backend-demand-controlled prefetch remain shared. Preserve SDK streaming for single, oversized or unknown-count manifests and custom/offline scans. ### Details - **Change map:** Production: 255 added / 156 removed lines, mostly extracting the existing parser into `IcebergManifestReader` and selecting load functions independently of cache policy. Tests: 158 added / 5 removed lines for both cache policies, snapshot/type/delete semantics, projection and fallback boundaries. Documentation: 12 added lines. No generated files. - **Why it was slow:** SDK ParallelIterable can sleep even after files are ready, and its refill/consumption cadence determines the number of fixed sleeps. - **Optimization:** Reuse completion-ordered prefetch and existing metadata scheduling without retaining uncached parsed contents. A request-local FileIO retains cancellation checks. Unfiltered scans without deletes reuse the native SDK column projection; cached values keep complete statistics for later predicates. Shared cache reads retain their existing ownership. - **Results:** With parsed Manifest Cache disabled, ARM64 full-SQL medians for 1000 manifests (10,000 small Parquet files) fall from 910.0 to 162.0ms for a cold single full scan, and 912.1 to 398.4ms for all four concurrent full scans to finish. Cold LIMIT falls 34.5 to 23.3ms. All 16 cold/warm LIMIT/full-scan medians are lower than the unchanged module. Cache-enabled full scans remain essentially flat; a 30-group followup measures 170.6->169.4ms for cold full scan, while warm LIMIT c4 remains 14.27->15.14ms (+0.88ms). Full-SQL medians in ms, parsed Manifest Cache disabled in both Doris versions: | Manifests | FE caches | SQL | Before, single | After, single | Before, 4 queries | After, 4 queries | |---:|---|---|---:|---:|---:|---:| | 100 | cold | LIMIT 10 | 34.7 | 25.7 | 39.5 | 29.7 | | 100 | cold | COUNT + SUM | 127.0 | 105.4 | 286.5 | 276.7 | | 100 | warm | LIMIT 10 | 23.2 | 12.0 | 28.9 | 16.1 | | 100 | warm | COUNT + SUM | 110.0 | 104.0 | 263.0 | 255.0 | | 1000 | cold | LIMIT 10 | 34.5 | 23.3 | 39.7 | 26.8 | | 1000 | cold | COUNT + SUM | 910.0 | 162.0 | 912.1 | 398.4 | | 1000 | warm | LIMIT 10 | 22.3 | 10.9 | 29.7 | 21.1 | | 1000 | warm | COUNT + SUM | 897.6 | 143.5 | 902.3 | 376.8 | Each cell is the median of 10 synchronized batches; the four-query columns measure completion of all four queries. Both versions use the same ARM64 Release BE, 6 CPUs per engine and the same 10,000 small files. Cold clears FE table/metadata/Manifest caches; BE footers and data pages stay warm. Warm retains the existing caches, with parsed Manifest caching still disabled. SQL/result caching and profiling are disabled during timing. - **Applicability and cost:** Native batch data scans with engine-owned scheduling and known manifest file counts fitting the bounded window. Single/large/unknown manifests retain SDK file streaming. Parsed-cache, synchronous COUNT, custom scan and wire-format behavior remain unchanged. - **Behavior change:** Manifest loading no longer depends on enabling the parsed-result cache for supported streaming scans. No new configuration is added. ### Testing - Native Linux ARM64 build.sh --fe -j2, including Checkstyle: passed. - Full Iceberg connector tests through run-fe-ut.sh on the final source: 1796 tests, 5 existing skips, no failures/errors. This includes cancellation, FileIO guards, completion ordering, backpressure, saturated scheduler admission and shared-load cancellation. - Native projection without mutating cached statistics; cache-enabled/disabled result equivalence and retention; partition type promotion; pinned historical/current snapshots; delete association; uncached load failure without SDK retry; oversized/unknown/overflow count guard. - Full-SQL comparisons: 1200 SQLs across old/new/SR, 800 SQLs with parsed caching enabled, and 300 focused followup SQLs. Both fixtures return 50000 rows and SUM(id)=1249975000; LIMIT returns 10 valid distinct ids. Same unchanged Release BE, 6 CPU per engine, swapped connector JAR only, profiling excluded from timing. Cold means FE metadata/manifest caches cleared while BE footer/pages are warm; SQL/result caching disabled. - Cache-off main order old/new/SR/SR/new/old; cache-on comparisons old/new/new/old. Warm both cached and cold-read paths before timing. All 14 measured stages have zero storage CPU throttling and zero host swap-out. - Separate 6 JFRs and 24 Profiles: 1000-manifest SDK hasNext+sleep samples 2053 before and 0 after; BE GetSplitTime 729/722ms before and 79/75ms after. All profiles have 10000 footer hits, 20000 page hits, 50000 rows and 10000 ranges, with no new footer reads or page writes. ### Release note Supported Iceberg batch scans use bounded completion notifications even when the parsed Manifest Cache is disabled, reducing fixed polling waits while retaining early stopping and Split-buffer backpressure. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No user-facing setting is added. Internal connector scheduling documentation is updated. - Pre-submission status: Ready. Local self-review, tests, build, package layout and controlled performance comparisons completed; remote CI has not run yet. The small cache-enabled warm-LIMIT c4 median difference is disclosed above. ### AI Disclosure - Tool: Codex - Generated: Code, tests, documentation and commit description - Human review: Pending -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
