xylaaaaa commented on issue #69884: URL: https://github.com/apache/doris/issues/69884#issuecomment-6682873512
### November 6 follow-up: repeated checkpoint reads during FE planning Read-only diagnosis at `fb4e695560bafc1862ac398fe22bfe6338c7f02e`, using the same local ASAN build and retained SF1 data as the [full TPC-DS measurement](https://github.com/apache/doris/issues/69884#issuecomment-6682219393). No production code, regression cases or expected outputs changed. The full-suite result remains **+4.69% overall**, not a new number derived from the following targeted probes. After all 24 Delta cardinalities matched the fixture manifest, ten alternating pairs of uninstrumented `EXPLAIN VERBOSE` produced: | Query | Scan nodes per reader | Hive median | Delta median | | --- | ---: | ---: | ---: | | Q88 | 32 | 27.873 ms | 324.592 ms | | Q80 | 18 | 29.337 ms | 247.685 ms | | Q22 | 3 | 5.137 ms | 45.496 ms | These measure planning plus plan-text transfer, without executing BE data scans. Separate query profiles also place the extra FE work in Translate: Hive/Delta **6/286 ms** for Q88 and **3/156 ms** for Q80, with no Delta planning GC in these samples. **Confirmed path:** the translator finalizes scan nodes sequentially; [DeltaLakeScanPlanProvider.planScan](https://github.com/apache/doris/blob/fb4e695560bafc1862ac398fe22bfe6338c7f02e/fe/fe-connector/fe-connector-deltalake/src/main/java/org/apache/doris/connector/deltalake/DeltaLakeScanPlanProvider.java#L282) opens a fresh [Kernel scan-file iterator](https://github.com/apache/doris/blob/fb4e695560bafc1862ac398fe22bfe6338c7f02e/fe/fe-connector/fe-connector-deltalake/src/main/java/org/apache/doris/connector/deltalake/DeltaLakeScanFiles.java#L60), which rereads the checkpoint through Hadoop S3A. The cached snapshot does not contain the active AddFile list or checkpoint bytes. - Q88 references four tables eight times each. Each of three traced EXPLAINs issued **160 checkpoint range GETs**, transferring **3,331,960 response-body bytes**. Repeating after all row counts became ready gave exactly the same counts and bytes. - Q80 issued **90 checkpoint GETs / 2,329,190 bytes** in each of three probes. A warm Hive Q88 control issued **zero MinIO requests** across ten EXPLAINs. - JFR and the deployed Kernel 4.4.1 bytecode confirm duplicate footer reads: one while preparing the Kernel reader and another when parquet-mr opens its reader. On these single-file checkpoints, two trailer/footer pairs plus a row-group read give **five range GETs per scan node**. This is FE checkpoint Parquet, not BE data Parquet. - In the warm Q88 trace, foreground MinIO socket waits averaged **135.0 ms/query**: footer 94.9 ms, checkpoint data/close 24.6 ms, and filesystem table-location checks 15.5 ms. Checkpoints account for **88.5% of that socket wait**, not 88.5% of total SQL time. Decoding, HTTP/Hadoop handling and other planner work remain. **Profile interpretation correction:** [Get Splits start/finish](https://github.com/apache/doris/blob/fb4e695560bafc1862ac398fe22bfe6338c7f02e/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java#L1005) overwrite one timestamp pair. The summary contains the last scan node's time, not the sum. Q88's 7 ms Get Splits therefore does not contradict its 286 ms Translate; split enumeration is included in Translate. No shared FE timer change was made. **Separate Q1 issue:** both readers still return the correct empty result. Delta prunes store to zero files but still reports RowsRead 100,000 / 55,820 / 366 for customer / store_returns / date_dim. Hive's RowsRead counters are all zero, although customer RawRowsRead is 32. Different join/build and CTE arrangements allow Hive to terminate earlier. A diagnostic sample shows wait/fetch **14/432 ms** for Hive/Delta; identifying the exact optimizer or execution rule needs a separate controlled-plan experiment. These single execution samples do not replace paired full-suite timings. Filesystem table-location HEAD/LIST is another, smaller foreground cost. A longer JFR also observed repeated location checks on the independent Statistics Job Appender thread; its time was not charged to the query. Next candidate: bounded reuse of small committed checkpoint content inside the Delta plugin, with identity/version/object correctness and large-table streaming preserved. This is a proposal, not an implemented cache or a promise to eliminate the entire 4.69% gap. END-03 remains open (63/67 tasks complete). Validation: official basic regression **36 result blocks passed**; Q1/Q88/Q80 actual results match across readers; all **96 retained TPC-DS objects** still match path/size/SHA-256. FE/BE and temporary Docker services are stopped, data retained. Local tasklist and the diagnosis report, full plans/profiles, raw JFR and sanitized request traces are preserved under `docs/delta-m1-preflight/benchmark/planning-diagnosis-2026-11-06/`. Initial JFR was taken before row-count refresh; its first trace-parser failure is explicitly retained as unavailable, with subsequent successful traces and warm-statistics verification kept separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
