goankur commented on PR #16145:
URL: https://github.com/apache/lucene/pull/16145#issuecomment-6083852932
@michaeljmarshall @uschindler here is a full matrix from one box and one set
of builds, so all numbers compare directly. Summary:
1. **Randomness is not needed.** Sampling by a hash of the page
(`BitMixer.mix64(offset >>> 12)`) performs the same as `ThreadLocalRandom`
(TLR) in every test below, and it is deterministic.
2. **The current head (dfc34c0) still slows hot term queries by about 20%.**
With the default `NO_FILES` preload, every file starts cold and probes its
first 1024 calls. If every file starts confident, the slowdown goes away. I
could not measure a cost from this under memory pressure, on a cold start, or
in your JMH suite.
**Setup.** g6.4xlarge (16 vCPU, 60 GiB RAM, local NVMe), kernel 6.18.51,
Corretto 25. All arms are built on main `3bb5f43990`, and only the store
classes differ:
| arm | what it is |
|---|---|
| main | main |
| no backoff | main with `-Dlucene.store.prefetchBackoff=false` (a
measurement-only switch) |
| PR head | dfc34c0 |
| confident + TLR | PR head, but the counter starts at N for every file |
| confident + hash | the same, with page-hash sampling instead of
`ThreadLocalRandom` |
### 1. 25M rerank, p99 ms
25M Cohere-v3 1024-d vectors, 1-bit BBQ + fp32 rerank of 500 candidates per
query (top-100 × overSample 5), fanout 100, single stream, 10k queries, 2 reps.
The memory limit is a systemd cgroup `MemoryMax`.
| | cgroup 10 GB (larger than RAM) | cgroup 20 GB (working set fits) |
|---|---|---|
| main | 172.4 / 172.4 | 131.0 / 132.2 |
| no backoff | 17.1 / 17.1 | 7.65 / 7.63 |
| PR head | 17.6 / 17.9 | 6.83 / 6.73 |
| confident + TLR | 17.5 / 17.2 | 6.79 / 6.68 |
| confident + hash | 17.5 / 17.7 | 6.84 / 6.90 |
Under memory pressure, starting confident costs nothing against the PR head,
and the hash's blind spot does not show. The hash variant also has the fewest
major faults (33 per query, against 38–45 for the PR head).
### 2. luceneutil wikimedium10m, QPS change vs main
6.1 GiB index. Each arm runs in 20 JVMs, alternating with the baseline, and
`fincore` confirms the index is 100% resident before each hot pair. bpftrace
counts `mincore` calls (the probes) per JVM.
**Hot** (about 55 query types; main against itself gives the noise floor):
| arm | AndHighLow | LowTerm | HighTermTitleSort | query types slower (p <
0.05) | query types faster (p < 0.05) | probes / JVM |
|---|---|---|---|---|---|---|
| main vs main | +0.5% | −0.8% | +0.8% | 2 | 0 | 395 |
| no backoff | −25.9% | −19.1% | −30.0% | 21 | 0 | 505k |
| PR head | −19.6% | −13.1% | −5.4% | 20 | 0 | 34.4k |
| confident + TLR | −1.9% | −1.9% | −2.9% | 0 | 0 | 5.0k |
| confident + hash | −0.1% | +3.2% | +2.5% | 1 | 4 | 2.6k |
**Cold start** (caches dropped before each pair). luceneutil starts the
second competitor's JVM first, so only that arm's first JVM sees the cold
cache. So I ran each cold pair twice: once with the arm taking the cold JVM,
and once with main taking it.
| arm | AndHighLow, arm takes cold JVM | AndHighLow, main takes cold JVM |
query types slower (p < 0.05), arm cold / main cold | query types faster (p <
0.05), arm cold / main cold |
|---|---|---|---|---|
| PR head | −19.9% | −16.7% | 21 / 25 | 0 / 0 |
| confident + TLR | −1.0% | −2.6% | 5 / 2 | 1 / 2 |
| confident + hash | −2.2% | −2.9% | 2 / 0 | 1 / 2 |
The PR head is about 17–20% slower in both orders. The two confident arms
stay within a few percent, and the direction changes with the order.
### 3. JMH kit
This is michaeljmarshall/lucene@e7138cc4 (neoremind's harnesses,
unmodified), run with `-f 1` in the hot, pressure, cold and hot-repeat
scenarios. The PR head and both confident arms are within about 2% of each
other on every row:
| scenario / benchmark | PR head | confident + TLR | confident + hash |
|---|---|---|---|
| hot prefetchOnly_T01 [topK=100] | −19.9% | −21.0% | −19.7% |
| hot prefetchOnly_T08 [topK=100] | −11.5% | −13.5% | −16.1% |
| hot retrievePrefetchThenRead_T08 [topK=100] | +2.6% | +4.0% | +2.9% |
| pressure mmapRandomBatchedPrefetch_T01 | +510% | +516% | +516% |
| pressure retrievePrefetchThenRead_T01 [topK=100] | +204% | +203% | +207% |
| pressure retrievePrefetchThenRead_T08 [topK=100] | −33.2% | −32.5% |
−32.8% |
| cold mmapRandomBatchedPrefetch_T01 | +4,148% | +4,138% | +4,215% |
| cold retrievePrefetchThenRead_T01 [topK=100] | +2,453% | +2,401% | +2,422%
|
Two results concern the PR as a whole, because all three arms show them. On
this box, hot `prefetchOnly` is −20% at 1 thread and −11% at 8 threads (on your
n2 box, T08 was +47%). Under pressure, `retrievePrefetchThenRead_T08` is −33%
(your run showed +7% and +20%, with wide error bars).
### Proposal
Keep N = 1024, SKIP = 64 and the reset on a miss, as in dfc34c0. Change two
things:
1. **Start every file confident, not only preloaded files.** A cold file is
then caught within about 64 calls, and it goes back to a probe on every call.
About the cost asymmetry: the cold-start pairs, the 10 GB rerank and the JMH
cold and pressure scenarios show no measurable cost.
2. **Sample by page hash.** It needs no state and no randomness, and the
result is the same in every run:
```java
private final AtomicInteger consecutiveHits = new AtomicInteger(N);
boolean shouldProbe(long offset) {
return consecutiveHits.get() < N || (BitMixer.mix64(offset >>> 12) & (SKIP
- 1)) == 0;
}
```
`prefetch()` passes its `offset`. The full core test suite passes with these
changes. The `slice(..., IOContext)` bounds check is fixed in dfc34c0.
**To reproduce:** the arms are branches in goankur/lucene: [main +
switch](https://github.com/goankur/lucene/tree/pr16145-eval/A-main), [PR
head](https://github.com/goankur/lucene/tree/pr16145-eval/K-pr-head),
[confident +
TLR](https://github.com/goankur/lucene/tree/pr16145-eval/G-confident-tlr),
[confident +
hash](https://github.com/goankur/lucene/tree/pr16145-eval/J-confident-hash).
Branch
[`pr16145-eval/results`](https://github.com/goankur/lucene/tree/pr16145-eval/results)
has the scripts, raw logs, bpftrace counts, JMH JSON, and `ARMS.txt` with the
commit SHA and lucene-core jar md5 of each arm.
**Not tested:** EBS or other network storage, concurrent query load, NRT
refresh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
