jimczi commented on issue #16044: URL: https://github.com/apache/lucene/issues/16044#issuecomment-4820469614
Thanks for digging into this @neoremind! I'm fairly sure these numbers are a benchmark bug, and I owe you an apology for not catching it earlier — more on that below. Starting from first principles: at **T01**, mmap + batched `WILLNEED` prefetch *should* beat everything. A single thread fires N async readaheads, the device services them in parallel (queue depth ~N), and the actual reads then land on pages that are already in flight — so one thread overlaps I/O the same way N threads would. Plain `pread` at T01 is stuck at queue depth 1, i.e. latency-bound. So mmap+prefetch losing to pread at T01 didn't add up — that's the kind of result that usually means something isn't doing what we think it is. Sure enough: the prefetch passes **unaligned** offsets to `posix_madvise`, and `madvise` requires a page-aligned start address — otherwise it returns `EINVAL` and does nothing. The return code is ignored, so the prefetch is a silent no-op. An `strace` of the JMH run on a c6id.4xlarge (NVMe) shows it plainly: ``` 5117 of 5181 madvise(0x…162f, 16384, MADV_WILLNEED) = -1 EINVAL (Invalid argument) ``` So every "mmap+prefetch" row was really just "mmap, no prefetch" = QD1. Page-aligning the madvise (rounding the start down to the page, exactly like `MemorySegmentIndexInput#advise` does) and rerunning cold (16K reads, page cache dropped, file > RAM): | 16K, all cold (ops/ms) | T01 | T08 | T16 | |---|---|---|---| | pread | 0.58 | 4.02 | 4.65 | | mmap, no prefetch (MADV_RANDOM) | 0.15 | 1.10 | 1.94 | | **mmap + prefetch (page-aligned)** | **4.19** | **5.62** | **5.98** | Before the fix that T01 cell was **0.15**; after, it's **4.19 ops/ms (~67k IOPS, ~1.25 GB/s)** — ~7× pread at T01, and right at the device's saturation ceiling (fio gives the same from a single thread at iodepth 16). Note that one prefetching thread (4.19) already matches pread at T08–T16 (4.0–4.7). The win is biggest at T01 and shrinks as you add threads, which is exactly what you'd expect: by T8/T16 the extra threads already supply the queue depth, so there's less for prefetch to add. Apologies for not flagging this sooner — I'd been running through the real `MMapDirectory`, which page-aligns before madvise, so it just worked for me and I never saw the `EINVAL` path. That's probably the main takeaway here: going through the actual Directory handles the alignment for you, so it's worth pointing the benchmark at it rather than re-deriving the madvise calls by hand. One more first-principles point: this parallel-prefetch trick isn't mmap-specific — you could get the same overlap from `pread` with async I/O / io_uring or a background prefetch thread. So I don't think the I/O-parallelism angle on its own justifies a native pread Directory; the `NativeThreadSet` contention you raised originally is the separate, real motivation and worth treating on its own. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
