neoremind commented on PR #16145: URL: https://github.com/apache/lucene/pull/16145#issuecomment-4989057175
I spent some time benchmarking and pressure-testing the prefetch path. Sharing the results here, hopefully these add some useful data and insights to the discussion. The key takeaways: 1. The backoff counter is two sides of the same coin. It's genuinely great when data is hot, genuinely harmful right at the mix-warm-cold memory pressure case (suppresses the prefetch we need most), and roughly a no-op when reads are mostly cold. 2. `isLoaded()` is stable and relatively cheap, worth keeping. It lets us skip the unnecessary `madvise` on already-resident pages, this shows advantage in hot/mixed workloads under concurrency workload, and it's cheap enough ~1us no matter hot/cold and concurrency level. 3. The real question isn't "keep the backoff or remove it", it's how we balance prefetch across hot and mixed warm/cold scenarios, a better backoff strategy could help here. Here are my rationales. The numbers below are based on 1. `RandomReadIOBenchmark` that I reuse from #16279, it measures backoff (current impl.) vs. [remove backoff but keep only `isLoaded()`](https://github.com/neoremind/lucene/commit/7b6d77c1638659986bdb18b26791a9bb558ec306) vs. [always-prefetch (what the PR promotes)](https://github.com/neoremind/lucene/blob/2bc9f931cd462fa67ce3c4ca298783ba492fbd3d/lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java#L330) vs. [customized slow-climb better backoff strategy](https://github.com/neoremind/lucene/commit/9bf1d1bc35b7631847d65d343933e8d0006c65c2), check out the code in the links. 2. I also tested a real-workload `StoredFieldsPrefetchBenchmark` to test top 100 field batch prefetch (16 per batch) and for-loop read, see code [here](https://github.com/neoremind/lucene/commit/13c55fe2e34efecdfbcbcfd13d85ea2ccdaadc33). 3. A micro benchmark on prefetch vs. `isLoaded()`, see code [here](https://github.com/neoremind/lucene/commit/9c6f59c1c17be60bce9a820e33333a95fa35e739). Platforms tested. | Platform | CPU | RAM | Storage | 4K random read latency (QD=1) | Saturated random read throughput | |---|---|---|---|---|---| | Linux EBS (EC2 c5.4xlarge) | 16 vCPU | 32G | EBS io2, 200G, 20K provisioned IOPS | ~334us | ~80 MB/s | | Linux NVME (EC2 c6id.4xlarge) | 16 vCPU | 32G | NVME SSD | ~103us | ~1.25 GB/s | *The latency and throughput numbers are calculated via `fio` with rand read 16k + different iodepth/numjobs to find the saturation point.* ### 1. When data is hot in the page cache, the backoff avoids prefetch overhead This is the case #13381 was built for and introduced the backoff counter. When everything is in RAM, every prefetch does an `isLoaded()`/`mincore` and/or a `madviseWillNeed()`, both are pure overhead when the pages already reside in RAM. The backoff counter is super useful to skip the needless overhead. Removing the backoff counter and `isLoaded()` = always-prefetch unconditionally, just adds cost on the prefetch path. The no-prefetch read should be the ceiling for hot warm scenario, look at below table for `RandomReadIOBenchmark`, the backoff sits right close to the ceiling while always-prefetch drops well below it: | | no prefetch | backoff (current impl.) | always-prefetch (this PR) | regression (always-prefetch vs. backoff) | |---|---|---|---|---| | EBS T1 | 40 | 40 | 27 | **-33%** | | EBS T16 | 300 | 299 | 179 | **-40%** | | NVMe T1 | 42 | 46 | 34 | **-26%** | | NVMe T16 | 310 | 314 | 217 | **-31%** | *(unit: ops/ms, higher is better)* With the backoff on, the prefetch path almost matches the plain-read, because prefetch is almost free due to counter climbs up in power-of-two increments when hot, backoff skips the `mincore`/`madvise` overhead to the maximum. That's the whole point of #13381. The real stored-fields workload tells the same story. I tested only on NVMe, top-k (k=100) store fields retrieval (batch prefetch on store fields to load 4K sized doc then for-loop read), index fully hot: | | no prefetch | backoff (current impl.) | always-prefetch (this PR) | regression (always-prefetch vs. backoff) | |---|---|---|---|---| | NVMe T1 | 2.86 | 2.66 | 1.95 | **-27%** | | NVMe T8 | 17.65 | 16.83 | 12.23 | **-27%** | ### 2. When most reads are cold, the backoff doesn't matter For a large portion of cold reads (say like >50%), removing backoff counter or not doesn't matter a lot, because page cache misses triggering page fault keep resetting the counter (`isLoaded() == false then set backoff counter.set(0)`), so it never climbs into large step "skip" territory. Prefetch is always ON beating no-prefetch version hard. Backoff does no harm for prefetch here. Here's the all-cold read scenario (64G file, page cache dropped each iteration) for `RandomReadIOBenchmark` run, backoff and always-prefetch are very close. | | no prefetch | backoff (current impl.) | always-prefetch (this PR)| regression | |---|---|---|---|---| | EBS T1 | 0.04 | 1.35 | 1.34 | ~0% | | EBS T16 | 0.26 | 1.36 | 1.37 | ~0% | | NVMe T1 | 0.15 | 2.69 | 2.91 | +8% | | NVMe T16 | 2.09 | 6.68 | 6.66 | ~0% | And the stored-fields workload at ~50% cold (64G index, ~half in RAM on Linux NVMe) says the same, backoff and always-prefetch are almost a tie, both ~6.5x better than no-prefetch read at T1 thanks for the batch async I/O firing upfront. Note that at T8, a bit surprising, I suspect the I/O queue is full from the 80K stored-fields blocks (default BEST_SPEED), so there's no spare parallelism left for prefetch to exploit. The prefetch effect saturates. | | no prefetch | backoff (current impl.) | always-prefetch (this PR) | |---|---|---|---| | NVMe T1 | 0.042 | 0.275 | 0.268 | | NVMe T8 | 0.321 | 0.286 | 0.266 | So, fully hot warm and heavily cold are both fine, the backoff neither helps nor hurts in the two scenarios. ### 3. At the mixed-warm-cold scenario, the backoff suppresses the prefetch it needs most This is where the PR attempts to address, in the mixed warm/cold, aggressive prefetch is much-needed, it attempts to load pages in ahead of the real read. Say in memory pressure scenario when ~90% of reads hit page cache and only ~10% are cold, misses are too rare to reset the backoff counter, it climbs the power-of-two ladder and suppresses prefetch on the cold pages that truly needed. Under memory pressure (index ≈ RAM), `RandomReadIOBenchmark`, backoff vs. always-prefetch: | | backoff (current impl.) | always-prefetch (this PR) | speedup | |---|---|---|---| | EBS T1 | 0.49 ops/ms | 2.27 ops/ms | **4.6× (+365%)** | | EBS T16 | 3.33 ops/ms | 7.11 ops/ms | **2.1× (+113%)** | | NVMe T1 | 2.16 ops/ms | 7.13 ops/ms | **3.3× (+230%)** | | NVMe T16 | 25.8 ops/ms | 60.4 ops/ms | **2.3× (+134%)** | Note that with prefetch, a single thread can already saturate the device (2.16 -> 7.13 ops/ms at T1 for NVMe), we don't need extra threads to pile on I/O to keep the queue deep. That's the real power of async prefetch to make I/O parallel even with a single thread. The stored-fields workload shows the same at T1: prefetch-then-read is **0.55 ops/ms** with always-prefetch vs just **0.19** with the backoff on (~2.9×). But the benefit diminishes as thread count increases. In the 16K × 16 `RandomReadIOBenchmark`, always-prefetch keeps winning even at T8/T16 (e.g. NVMe 43.8 vs 14.9 for backoff), whereas, stored fields is not: at T8 always-prefetch (0.99) actually lags backoff current impl (1.32). Like described above, the reason could be stored fields default to 80K block (`BEST_SPEED` strategy), so a "4K doc x 100" op is really ~100 scattered 80K block reads. Those big-block reads could saturate the device at much lower concurrency, leaving no room for async I/O prefetch to exploit. Anyway, the T1 in store fields case already proves the story that always-prefetch aggressively is better than backoff in mixed-warm-cold. ### 4. `isLoaded()` is a cheap and stable guard worth keeping Zoom in `isLoaded()`, I ran a micro on the NVMe (c6id.4xlarge, 4K page), aligned the start address for `madvise` to ensure prefetch did work, compare apples-to-apples: | | T1 | T8 | T1 -> T8 scaling | |---|---|---|---| | `isLoaded()` / mincore, warm | 0.57us | 1.28us | ~2× | | `isLoaded()` / mincore, cold | 0.43us | 1.25us | ~2× | | `madvise(WILLNEED)`, warm | 0.33us | 2.5us | ~7× | | `madvise(WILLNEED)`, cold | 5.3us | 42us | ~8× | `isLoaded()` is cheap and stable, and it scales only ~2× from T1->T8. `madvise(WILLNEED)` latency is lower than `isLoaded()` single-threaded when pages are warm (0.33us madvise vs. 0.57us `isLoaded()` check). But it's no free lunch, counter-intuitively, it's contention-prone, madvise on warm is 2x the latency of `isLoaded()` at T8 (2.5us vs. 1.28us), and its latency climbs ~7–8× with threads and blows up on cold pages (the async WILLNEED hint isn't free, still has to submit and queue the I/O). So, in terms of `isLoaded()`'s value: it saves the `madvise` on warm pages which is a bit cheap single-threaded, but it contended and expensive under concurrency. To be fair, I admit that prefetch in real use case usually interleaves with other computations, so the contention may not be as severe as in this tight micro-benchmark loop. Looking benchmarks at Linux NVMe for hot and mixed-hot-cold scenario, keeping `isLoaded()` check is better than always-prefetch except the acceptable loss in single threaded: - `RandomReadIOBenchmark`, hot: **33.3 / 224.1 / 256.6** (keep `isLoaded`) vs **34.0 / 169.4 / 216.7** (always-prefetch), T1/T8/T16 - stored fields, hot: **2.14 / 13.86** (keep `isLoaded`) vs **1.95 / 12.23** (always-prefetch), T1/T8 - `RandomReadIOBenchmark`, mixed hot-warm: **6.55 / 44.9 / 61.2** (keep `isLoaded`) vs **7.13 / 43.8 / 60.4** (always-prefetch), T1/T8/T16 - stored fields, mixed hot-warm: **0.56 / 1.0** (keep `isLoaded`) vs. **0.55 / 0.99** (always-prefetch), T1/T8 Looking benchmarks at Linux NVMe for cold scenario. The extra `isLoaded()` is definitely overhead, but compare with EBS random-read latency ~340us, normal SSD ~100us, a performant NVMe 10–20us, the guard latency looks negligible relative to end-to-end I/O overall. ### 5. My 2 cents: a better backoff instead of removing it, and keep isLoaded() So the problem really isn't "backoff yes/no", removing it is great for mixed warm/cold but bad for hot, there is no one-size-fits-all. I think the power-of-two backoff strategy is just a bit too aggressive, what if we soften it to balance hot vs. mixed hot-warm reads? I tried a [**slow-climbing backoff**](https://github.com/neoremind/lucene/commit/9bf1d1bc35b7631847d65d343933e8d0006c65c2), fire madvise for the first 8 prefetch, then every Nth with N growing slowly, cap at 64 to force a prefetch. Tested on `StoredFieldsPrefetchBenchmark` (Linux NVMe, prefetch-then-read ops/ms, higher = better): | | thread | no prefetch | backoff (current impl.) | always-prefetch (this PR) | slow-climb (customized backoff) | |---|---|---|---|---|---| | Hot 16G | T1 | 2.86 | 2.66 | 1.95 | 2.37 | | Hot 16G | T8 | 17.65 | 16.83 | 12.23 | 15.15 | | Boundary 32G (≈RAM) | T1 | 0.15 | 0.19 | 0.55 | 0.47 | | Boundary 32G (≈RAM) | T8 | 1.30 | 1.32 | 0.99 | 1.53 | In the hot rows, slow-climb sits just under the no-prefetch ceiling (only -9%, vs always-prefetch's -26%), nearly as cheap as the current backoff. At the boundary, T1 it does ~3.0x over its no-prefetch baseline (vs 1.4x for the current power-of-two backoff and ~3.7x for always-prefetch which is the best), a somewhat good middle ground. We could explore a softer better backoff strategy alongside the `isLoaded()` guard, like I described, this is really a balance. ## Appendix — raw results Columns across all tables: - **backoff (current impl.)** = current (#13381) - **rm-backoff** = remove backoff counter but keep `isLoaded()` - **always-prefetch** = remove both (prefetch unconditionally) - **slow-climb** = my customized backoff. <details> <summary>1. RandomReadIOBenchmark (reused from #16279) — RANDOM + batched prefetch, ops/ms (higher = better)</summary> Numbers under test of `mmapRandomBatchedPrefetch` (RANDOM advice, prefetch the 16 random offsets then read). `readSize=16K`, `readsPerOp=16`. **Linux EBS (c5.4xlarge)** <img width="830" height="505" alt="Screenshot 2026-07-16 at 2 30 42 PM" src="https://github.com/user-attachments/assets/61b413a2-57e4-4592-831f-ddcd4c679ae8"; /> | Case | Threads | backoff (current impl.) | rm-backoff | always-prefetch | slow-climb | |---|---|---|---|---|---| | Warm 16G (hot) | T1 | 39.98 | 25.36 | 26.95 | 35.35 | | Warm 16G | T4 | 151.18 | 96.67 | 85.65 | 130.40 | | Warm 16G | T8 | 272.70 | 180.33 | 129.56 | 243.83 | | Warm 16G | T16 | 298.66 | 218.23 | 179.06 | 279.37 | | Mem pressure 32G (≈RAM) | T1 | 0.49 | 2.29 | 2.27 | 0.76 | | Mem pressure 32G | T4 | 1.98 | 7.21 | 7.19 | 3.21 | | Mem pressure 32G | T8 | 3.30 | 7.18 | 7.24 | 4.33 | | Mem pressure 32G | T16 | 3.33 | 7.16 | 7.11 | 4.37 | | Many cold 64G (no drop) | T1 | 0.64 | 1.58 | 1.52 | 1.63 | | Many cold 64G | T4 | 1.50 | 1.65 | 1.66 | 1.74 | | Many cold 64G | T8 | 1.50 | 1.64 | 1.65 | 1.71 | | Many cold 64G | T16 | 1.47 | 1.64 | 1.65 | 1.70 | | All cold 64G (drop cache) | T1 | 1.35 | 1.35 | 1.34 | 1.35 | | All cold 64G | T4 | 1.35 | 1.35 | 1.35 | 1.35 | | All cold 64G | T8 | 1.35 | 1.35 | 1.35 | 1.35 | | All cold 64G | T16 | 1.36 | 1.36 | 1.37 | 1.37 | **Linux NVMe (c6id.4xlarge)** <img width="764" height="454" alt="Screenshot 2026-07-16 at 2 30 55 PM" src="https://github.com/user-attachments/assets/490f284c-aa94-41ab-92ad-1ffca619dee5"; /> | Case | Threads | backoff (current impl.) | rm-backoff | always-prefetch | slow-climb | |---|---|---|---|---|---| | Warm 16G (hot) | T1 | 45.75 | 33.27 | 34.02 | 41.44 | | Warm 16G | T4 | 170.53 | 122.45 | 111.72 | 154.45 | | Warm 16G | T8 | 310.24 | 224.10 | 169.38 | 283.71 | | Warm 16G | T16 | 313.69 | 256.63 | 216.72 | 315.65 | | Mem pressure 32G (≈RAM) | T1 | 2.16 | 6.55 | 7.13 | 3.36 | | Mem pressure 32G | T4 | 8.14 | 25.01 | 23.81 | 12.21 | | Mem pressure 32G | T8 | 14.88 | 44.91 | 43.84 | 22.35 | | Mem pressure 32G | T16 | 25.78 | 61.21 | 60.38 | 38.13 | | Many cold 64G (no drop) | T1 | 2.61 | 3.11 | 3.16 | 3.09 | | Many cold 64G | T4 | 7.21 | 7.21 | 7.18 | 7.21 | | Many cold 64G | T8 | 7.21 | 7.21 | 7.18 | 7.21 | | Many cold 64G | T16 | 7.21 | 7.21 | 7.18 | 7.19 | | All cold 64G (drop cache) | T1 | 2.69 | 2.70 | 2.91 | 2.68 | | All cold 64G | T4 | 6.24 | 6.21 | 6.24 | 6.24 | | All cold 64G | T8 | 6.28 | 6.27 | 6.27 | 6.29 | | All cold 64G | T16 | 6.68 | 6.67 | 6.66 | 6.65 | Baseline `mmapRandom` (no prefetch) is constant baseline across branches, EBS T16 ~300 ops/ms warm / ~3.3 mem-pressure, NVMe T16 ~310 ops/ms / ~26 ops/ms. </details> <details> <summary>2. Micro: isLoaded() vs madvise(WILLNEED), page-offset aligned, us/op (lower = better)</summary> | Platform | Case | Threads | readSize | isLoaded() | madvise(WILLNEED) | |---|---|---|---|---|---| | EBS | warm | T1 | 4K | 1.114 | 0.676 | | EBS | warm | T1 | 16K | 1.448 | 0.743 | | EBS | warm | T8 | 4K | 1.712 | 2.940 | | EBS | warm | T8 | 16K | 1.810 | 3.097 | | EBS | cold | T1 | 4K | 0.805 | 47.467 | | EBS | cold | T1 | 16K | 0.863 | 46.521 | | EBS | cold | T8 | 4K | 1.306 | 390.060 | | EBS | cold | T8 | 16K | 1.334 | 379.543 | | NVMe | warm | T1 | 4K | 0.572 | 0.326 | | NVMe | warm | T1 | 16K | 0.642 | 0.378 | | NVMe | warm | T8 | 4K | 1.282 | 2.515 | | NVMe | warm | T8 | 16K | 1.252 | 2.500 | | NVMe | cold | T1 | 4K | 0.429 | 5.351 | | NVMe | cold | T1 | 16K | 0.454 | 13.849 | | NVMe | cold | T8 | 4K | 1.254 | 42.509 | | NVMe | cold | T8 | 16K | 1.215 | 109.740 | </details> <details> <summary>3. StoredFieldsPrefetchBenchmark (NVMe) — top-100 retrieval, ops/ms (higher = better)</summary> Real workload: collect top-100 hits, prefetch a window of 16, then retrieve. `docSizeBytes=4096`, RANDOM advice. Note stored fields use an 80KB BEST_SPEED block by default, so each doc read faults/decompresses a large block, not 4KB. **Hot (16G index, fully cached)** | Method | Threads | backoff (current impl.) | rm-backoff | always-prefetch | slow-climb | |---|---|---|---|---|---| | retrieveNoPrefetch | T1 | 2.86 | 2.66 | 2.64 | 2.61 | | retrieveNoPrefetch | T8 | 17.65 | 16.13 | 16.33 | 16.27 | | prefetchThenRead | T1 | 2.66 | 2.14 | 1.95 | 2.37 | | prefetchThenRead | T8 | 16.83 | 13.86 | 12.23 | 15.15 | **Memory pressure (32G ≈ RAM)** | Method | Threads | backoff (current impl.) | rm-backoff | always-prefetch | slow-climb | |---|---|---|---|---|---| | retrieveNoPrefetch | T1 | 0.134 | 0.150 | 0.150 | 0.156 | | retrieveNoPrefetch | T8 | 1.165 | 1.255 | 1.296 | 1.307 | | prefetchThenRead | T1 | 0.191 | 0.558 | 0.554 | 0.474 | | prefetchThenRead | T8 | 1.322 | 1.007 | 0.986 | 1.534 | **50% cold (64G, ~half resident)** | Method | Threads | backoff (current impl.) | rm-backoff | always-prefetch | slow-climb | |---|---|---|---|---|---| | retrieveNoPrefetch | T1 | 0.042 | 0.040 | 0.042 | 0.042 | | retrieveNoPrefetch | T8 | 0.321 | 0.310 | 0.321 | 0.320 | | prefetchThenRead | T1 | 0.275 | 0.259 | 0.268 | 0.270 | | prefetchThenRead | T8 | 0.286 | 0.259 | 0.266 | 0.267 | </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
