jimczi commented on issue #16044:
URL: https://github.com/apache/lucene/issues/16044#issuecomment-4766553900
Spent a bit more time on the prefetch angle, since I think the "prefetch
doesn't help" result for random reads is really about how it's issued, not
about prefetch.
A prefetch is fire-and-forget — it starts the I/O and returns. So
prefetching a block and then immediately reading it gains nothing: you block on
the read before the I/O lands. To get any benefit you have to start several
I/Os for *different* blocks and *then* read them, so they're in flight together
— a queue depth of N from a single thread. The random arm does the opposite:
`doMmapMadvRandomWillneedReads` prefetches a block and reads it in the same
loop iteration (queue depth 1). Your own 4 KB numbers show it doing nothing —
at 64 G, `MADV_RANDOM + WILLNEED` = 1.15 == `MADV_RANDOM` alone = 1.15, both
below pread (1.96). @neoremind basically described this in observation #2.
The fix is to split the loop: prefetch the whole batch of offsets first,
then read them.
<details>
<summary>diff for
<code>RandomReadIOBenchmark#doMmapMadvRandomWillneedReads</code></summary>
```diff
private void doMmapMadvRandomWillneedReads(ThreadBuffers tb, Blackhole
bh) {
ThreadLocalRandom rng = ThreadLocalRandom.current();
byte[] dst = tb.heapBuf.array();
+ long[] offsets = new long[READS_PER_OP];
try {
- for (int i = 0; i < READS_PER_OP; i++) {
- long offset = rng.nextLong(MAX_OFFSET);
- MemorySegment slice = mmapSegmentMadvRandom.asSlice(offset,
READ_SIZE);
- int rc = (int) POSIX_MADVISE.invokeExact(slice, (long) READ_SIZE,
MADV_WILLNEED);
- MemorySegment.copy(mmapSegmentMadvRandom, ValueLayout.JAVA_BYTE,
offset, dst, 0, READ_SIZE);
- bh.consume(dst[0]);
- }
+ for (int i = 0; i < READS_PER_OP; i++) {
+ offsets[i] = rng.nextLong(MAX_OFFSET);
+ }
+ // 1) hand the kernel all the ranges up front -> READS_PER_OP I/Os in
flight
+ for (int i = 0; i < READS_PER_OP; i++) {
+ MemorySegment slice = mmapSegmentMadvRandom.asSlice(offsets[i],
READ_SIZE);
+ int rc = (int) POSIX_MADVISE.invokeExact(slice, (long) READ_SIZE,
MADV_WILLNEED);
+ }
+ // 2) then block on the reads; they overlap
+ for (int i = 0; i < READS_PER_OP; i++) {
+ MemorySegment.copy(mmapSegmentMadvRandom, ValueLayout.JAVA_BYTE,
offsets[i], dst, 0, READ_SIZE);
+ bh.consume(dst[0]);
+ }
} catch (Throwable t) {
throw new RuntimeException(t);
}
}
```
</details>
I couldn't run on EBS, so here's the shape on my laptop (Apple-Silicon NVMe,
64 GB file > RAM, page cache dropped before each run, 16 KB reads, single
thread, plain `NORMAL` mapping). Treat absolutes as relative — the point is the
per-block vs batched gap:
- serial mmap: ~170 MB/s
- serial pread: ~210 MB/s
- prefetch one block then read it (current arm): ~220 MB/s — no gain
- prefetch the batch of 16 first, then read: ~1.4 GB/s (qd 128: ~2.7 GB/s,
device ceiling)
And with threads — pread scales (one in-flight each), but a single
batched-prefetch thread keeps up with pread at 4–8 threads, and 4 threads
saturate the disk:
| MB/s | T1 | T4 | T8 |
|---|---|---|---|
| pread | 285 | 1130 | 1880 |
| mmap + batched prefetch (qd 16) | 1550 | 3450 | (ceiling) |
On EBS I'd expect batched prefetch to at least reach the pread line, since
the device (not the read path) is the ceiling there.
A few things worth separating out:
- This is on `NORMAL`, the default — `RANDOM` is only set on a handful of
codec files, so the default is the interesting case. `WILLNEED` loads exactly
the ranges you pass it, so you read only what you need without `MADV_RANDOM`;
that's an extra lever for purely-random files, not a prerequisite.
- The "sequential" arm is actually random access too: each op jumps to a
fresh random offset and reads a contiguous 64 KB chunk (16 × 4 KB) instead of
16 scattered 4 KB pages — so it's `random 4 KB` vs `random 64 KB`, and never
sustains a real sequential stream, which is why the fixed 64 KB WILLNEED there
does nothing. A real sequential workload is a whole-file scan on a file > RAM.
I ran one (cold 32 GB, single thread): RANDOM mmap with a sliding 2 MB prefetch
window ≈ 2.5 GB/s, matching uncached pread (≈ 2.4 GB/s) and the kernel's
SEQUENTIAL readahead (≈ 2.6 GB/s), vs ≈ 0.65 GB/s with no prefetch. So mmap
keeps pace on real sequential too — you'd let `ReadAdvice.SEQUENTIAL`/kernel
readahead handle it, or use a sliding window (what merge does). Might be worth
swapping `SequentialReadIOBenchmark` for an actual whole-file scan.
- mmap reads are zero-copy and have no per-read lock, so they don't hit the
`NativeThreadSet` contention you measured for `FileChannel`.
This batch-then-read shape is exactly what Lucene's pure-random paths do (or
are meant to), so it's not a hypothetical for codecs:
- top-N stored fields: `StoredFields#prefetch(docID)` — collect the top-N
hits, prefetch them all, then retrieve.
- KNN exact rescoring: `KnnVectorValues#prefetch(int[] ords, int numOrds)` /
`PrefetchableFlatVectorScorer` — prefetch the candidate vectors, then score.
The hooks exist; they're just not wired into the default search loop, so
today it's driven app-side (ES/OpenSearch).
Put together, mmap looks competitive across the board:
- random, in-memory: far ahead of pread (your 16 G: 1025 vs 157)
- random, cold/> RAM: matches or beats pread once the prefetch is batched,
and saturates the disk with fewer threads
- sequential: ties pread on a real whole-file scan
plus zero-copy and no per-read lock.
Which brings me back to the original report — what was the workload where
`pgmajfault` spiked once the working set exceeded the cgroup limit? If we can
pin that down, I suspect it's fixable on the mmap side (e.g. wiring prefetch
into the HNSW search path, or `MADV_RANDOM` on the hot files) rather than by
switching the read mechanism — and I'd be happy to help dig into it. The one
motivation that's genuinely about the read mechanism is the `NativeThreadSet`
monitor in `FileChannelImpl` — that's a real `NIOFSDirectory` scaling ceiling
and a fair reason for an FFI pread path there, but it's a `FileChannel`
limitation, not an argument against mmap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
