xylaaaaa commented on issue #69884: URL: https://github.com/apache/doris/issues/69884#issuecomment-6680395190
Progress update — Delta-only small-table scheduling (2026-11-06) Implemented and locally committed as `c571356e077e209e95b5fa1ec0c0954ea1fb51f3` on `codex/delta-m1-reader`. This commit has **not been pushed** to PR #69904 yet. The change is confined to four production files in the Delta plugin, plus its tests and README. With no current CRC, scheduling reuses a complete count from the statement or the same user/table identity/version statistics-cache entry. Existing statistics collection counts files; exhaustive normal scans can also publish their already-observed total without starting a statistics task. This is necessary because FE background row-count sessions and SQL sessions can have different identities; user isolation is preserved. A late report only updates its captured entry and cannot republish an invalidated entry. There are no shared FE/BE, SPI, or wire-format changes. The scheduling gate does not enumerate files, submit work, or wait for statistics. A known small count selects synchronous planning; unknown and large counts keep streaming. Cold snapshots without CRC may still stream. Retained-data comparison, using the same local macOS ARM64 ASAN BE: a two-file Delta table, one selected file, one returned decimal value (`123.45`), SQL cache disabled, five warmup pairs and sixteen alternating measured pairs per entrance. Profiles were collected separately from timing. | Entrance | Before automatic median | After automatic median | Reduction | Forced-sync control before / after | | --- | ---: | ---: | ---: | ---: | | Filesystem | 61.614 ms | 41.420 ms | 32.8% | 35.978 / 41.056 ms | | Dedicated HMS | 66.370 ms | 38.509 ms | 42.0% | 44.302 / 37.627 ms | | HMS gateway | 57.911 ms | 32.874 ms | 43.2% | 32.914 / 32.370 ms | All three automatic paths went from **144 scanners to one**, with the same one file/split/row read and correct result. The final automatic and forced-sync medians differ by 0.4–0.9 ms. These are warm, small-table measurements; the controls also vary across runs, and this is **not a new TPC-DS performance result**. Large-table guard: the retained million-file checkpoint was tested before/after in separate **512 MiB** JVMs. All three runs enumerated exactly 1,000,000 ranges; enumeration medians were 1493.563 / 1451.023 ms. Both completed background statistics and a complete count reused by subsequent queries still selected streaming. Sampled scan heap peaks were 133,327,520 / 238,172,480 bytes: the bounded-heap tests passed, but this does not establish unchanged memory use. The fixture contains planning metadata, not the corresponding million data files, so this does not measure full data IO throughput. Validation: - Full `./build.sh --fe -j4` passed, including Checkstyle; the ASAN BE binary was unchanged. - 92 regular Delta/Kernel/metastore tests plus two separately enabled million-file tests passed (94 actual tests). The optional diagnostic PlanningMetrics test was not run. - Basic, statistics, CAST pruning, Parquet DV, mixed HMS, and planning regressions passed: **6 suites, 129 result blocks**. All 14 existing Delta p0/p2 golden files retained their hashes; no `.out` was edited or regenerated. - The planning regression initially pointed at the wrong local MinIO and could not find `delta_bench`; it passed after selecting the retained fixture's original endpoint, without changing the test/data. README, local design notes and tasklist are updated. Raw samples, profiles, test logs and the checkpoint-by-checkpoint self-review are retained in the worktree under `docs/delta-m1-preflight/small-table-scheduling-20261106/`. Test services are stopped and data is preserved. Full TPC-DS parity remains open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
