neoremind commented on issue #16044: URL: https://github.com/apache/lucene/issues/16044#issuecomment-4810945398
Thank you @jimczi for the detailed feedback. I revised the JMH benchmark to use "prefetch all offsets first, then read" as suggested, see updated code in https://github.com/apache/lucene/pull/16279. Reran and here's what I found. ## Platforms | Platform | CPU | RAM | Storage | 4K random read latency (QD=1) | Saturated random read throughput | |---|---|---|---|---|---| | Mac (M3 Pro) | 12 CPU | 36G | 512G NVME SSD | ~77us | ~840 MB/s | | Linux EBS (EC2 c5.4xlarge) | 16 vCPU | 32G | EBS io2, 20K provisioned IOPS | ~334us | ~80 MB/s | | Linux NVME (EC2 c6id.4xlarge) | 16 vCPU | 32G | NVME SSD | ~103us | ~1250 MB/s | The latency and throughput numbers are calculated via `fio` with rand read 16k + different iodepth/numjobs to find the saturation point. ## Random Read Results 16KB reads (4 pages) x 16 reads/op = 256KB per op. Throughput = ops/ms x 256 KB. For example, with O_DIRECT on c6id.4xlarge (NVME), JMH at 16 threads results in 4.9 ops/ms, that's `4.9 x 1000 x 256KB ~= 1250 MB/s`, matching the peak random read throughput from fio. I removed the O_DIRECT results from below tables to focus on pread, FileChannel, and mmap. Showing **T**hread1 and **T**hreads 8 only, full data can be expanded in Appendix. ### CASE 1: Warm 16G file fits in RAM, pre-warmed with `cat file > /dev/null`. | Strategy | Mac T1 | Mac T8 | EBS T1 | EBS T8 | EC2 NVME T1 | EC2 NVME T8 | |---|---|---|---|---|-------------|-------------| | mmap (NORMAL) | 78 | **285** | 40 | **267** | 47 | **322** | | mmap (MADV_RANDOM) | 78 | **285** | 39 | **268** | 46 | **323** | | mmap + batchedPrefetch (NORMAL) | 61 | **204** | 30 | **209** | 42 | **294** | | mmap + batchedPrefetch (RANDOM) | 61 | **197** | 30 | **205** | 43 | **294** | | FFI pread | 38 | **76** | 25 | **153** | 32 | **209** | | FileChannel | 37 | **73** | 24 | **93** | 30 | **92** | ### CASE 2: Memory pressure 32G file > ~29G available RAM (JVM takes 2G), pre-warmed. | Strategy | Mac T1 | Mac T8 | EBS T1 | EBS T8 | EC2 NVME T1 | EC2 NVME T8 | |---|---|---|---|---|-------------|-------------| | mmap (NORMAL) | 5.8 | **24** | 0.91 | **3.8** | 2.8 | **18** | | mmap (MADV_RANDOM) | 5.3 | **29** | 0.54 | **3.7** | 2.1 | **15** | | mmap + batchedPrefetch (NORMAL) | 7.2 | **35** | 0.96 | **4.7** | 3.2 | **16** | | mmap + batchedPrefetch (RANDOM) | 8.1 | **33** | 0.54 | **3.6** | 2.2 | **15** | | FFI pread | 2.6 | **13** | 1.4 | **5.9** | 4.1 | **27** | | FileChannel | 2.0 | **12** | 1.2 | **5.3** | 3.7 | **25** | ### CASE 3: Many cold reads 64G file (2x available RAM), ~50% cold reads, pre-warmed. | Strategy | Mac T1 | Mac T8 | EBS T1 | EBS T8 | EC2 NVME T1 | EC2 NVME T8 | |---|---|---|---|---|-------------|-------------| | mmap (NORMAL) | 0.46 | **2.7** | 0.13 | **0.51** | 0.45 | **1.2** | | mmap (MADV_RANDOM) | 0.44 | **3.3** | 0.072 | **0.46** | 0.28 | **2.0** | | mmap + batchedPrefetch (NORMAL) | 1.19 | **4.4** | 0.13 | **0.52** | 0.46 | **1.2** | | mmap + batchedPrefetch (RANDOM) | 1.14 | **4.4** | 0.071 | **0.45** | 0.28 | **2.0** | | FFI pread | 0.53 | **4.3** | 0.24 | **1.70** | 0.54 | **3.9** | | FileChannel | 0.78 | **4.2** | 0.21 | **1.43** | 0.53 | **3.8** | ### CASE 4: Almost all cold reads 64G file (2x available RAM), clear page cache before each iteration, cold start. | Strategy | Mac T1 | Mac T8 | EBS T1 | EBS T8 | EC2 NVME T1 | EC2 NVME T8 | |---|---|---|---|---|-------------|-------------| | mmap (NORMAL) | 0.32 | **2.0** | 0.072 | **0.32** | 0.27 | **0.87** | | mmap (MADV_RANDOM) | 0.20 | **1.9** | 0.038 | **0.26** | 0.15 | **1.16** | | mmap + batchedPrefetch (NORMAL) | 0.42 | **2.8** | 0.071 | **0.32** | 0.27 | **0.87** | | mmap + batchedPrefetch (RANDOM) | 0.43 | **2.8** | 0.038 | **0.26** | 0.15 | **1.16** | | FFI pread | 0.45 | **2.7** | 0.13 | **1.08** | 0.29 | **2.27** | | FileChannel | 0.40 | **2.7** | 0.13 | **1.08** | 0.29 | **2.27** | ## My takes ### 1. Warm page cache: mmap is the big winner When everything is in RAM, mmap removes syscall overhead, it's pure page-table lookup with direct pointer to cached page, also no intermediate offheap buffer copy like FileChannel. At T8, mmap is ~4 faster than pread on Mac, and 1.5-2x faster on Linux. Observed that batched WILLNEED prefetch adds ~25–30% overhead since the posix_madvise syscall is not necessary when pages already reside in RAM. ### 2. Under memory pressure and cold reads on Linux, mmap's page-fault is expensive Under memory pressure (cases 2,3,4), on Mac, mmap with batched prefetch on mmap beats pread under moderate pressure and matches pread under severe cold pressure. This aligns with your finding that mmap + prefetch is competitive on fast NVME on Mac. However, the story is different on Linux, both EBS and NVME, pread outperforms all mmap variants under cold pressure. The more cold reads there are, the higher concurrency, the bigger pread's advantage becomes. My understanding of why: mmap's page fault is more expensive than pread syscall on Linux, especially with memory pressure. A page fault involves: trap into kernel, virtual memory address walk, page-table allocation/update, I/O submission, wait, map page, return to userspace, things get even heavier with memory reclaim path. While pread does syscall, submit I/O, wait, copy to buffer, return. Interestingly, [this paper from Andy Pavlo](https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf) argues mmap is not the best option for larger-than-memory DBMS workloads if not used carefully. It notes InfluxDB, MongoDB, and SingleStore deprecated mmap in their latest releases. Section 3.4 (Problem #4: Performance Issues) points to OS page eviction mechanisms that don't scale beyond a few threads for larger-than-memory workloads on high-bandwidth storage with bottlenecks like page table contention, page eviction overhead, and TLB shootdowns. (There's [a counter-argument](https://ravendb.n et/articles/re-are-you-sure-you-want-to-use-mmap-in-your-database-management-system) worth reading for balance.) I know mmap in DB is a controversial topic, and it deserves more debate than I can give it here, so noting it just for reference. I didn't dive deeper due to time bandwidth, but the benchmark data does correlate with what the paper describes: for larger-than-memory files on Linux, mmap is powerful when pages are warm and memory is sufficient, but under cold-read pressure there is intrinsic disadvantages in page-fault path that pread doesn't have. ### 3. Normal vs Random hint Zooming into cases 3 and 4 where I/O dominates on Linux. I agree with you that Normal fits most cases, and Random should be selectively chosen based on the workload. The right choice comes down to the factors in **memory pressure**, **multi-threaded load** and **I/O latency**. In these memory-constrained random read scenarios with high page eviction pressure, for slow device like EBS (~334us latency), Normal's readahead amortizes that cost by loading multiple pages remotely, better to over-fetch, so we can see Normal outperforms Random a lot in EBS. While for faster device like NVME, especially at T8 where eviction pressure is higher due to more threads, readahead becomes counter-productive loading both useful or unnecessary pages that may get evicted soon, this wastes IOPS, so Random preserves IOPS avoiding polluting page cache, so we can see Random wins with NVME SSD on Linux. ### 4. Batched prefetch or not In cases 3 and 4 on Linux, the batched WILLNEED prefetch add no measurable benefit, the same with my earlier findings. Like you pointed out, this is really about how it's issued and tested in my JMH benchmarks: in a tight loop with almost no computation gap, the async I/O can't complete before the read faults. In real-world use case, prefetch should be genuinely helpful. In detail, in JMH, the 16 `posix_madvise(WILLNEED)` calls submit async I/O for 16 offsets, but the subsequent 16 `MemorySegment.copy` calls fault immediately. The async I/O hasn't completed in that time window. Think about ~334us in EBS, impossible to complete. Then the fault handler submits the same I/O again (or finds it already in-flight). I think prefetch only helps when there's **interleaving** between the hint and the read like CPU intensive computations occur right after prefetch before read. Thanks for pointing out the code in paths like `StoredFields#prefetch`, `KnnVectorValues#prefetch` / `PrefetchableFlatVectorScorer`, these usage offers overlap naturally: ```java // PrefetchableFlatVectorScorer — computation overlaps with prefetch I/O @Override public float bulkScore(int[] nodes, float[] scores, int numNodes) throws IOException { values().prefetch(nodes, numNodes); // fire async I/O for all candidates return delegate.bulkScore(nodes, scores, numNodes); // compute while I/O completes } @Override public void prefetch(final int[] ordsToPrefetch, int numOrds) throws IOException { if (ordsToPrefetch == null) return; int finalNumOrds = Math.min(numOrds, ordsToPrefetch.length); if (finalNumOrds <= 1) return; for (int i = 0; i < finalNumOrds; i++) { long offset = (long) ordsToPrefetch[i] * byteSize; slice.prefetch(offset, byteSize); } } ``` And in `StoredFields`, callers decide when to call prefetch and can overlap it with their own logic. ### 5. Practical thoughts For deployments with larger-than-memory workloads on Linux, pread provides a measurable advantage in the pure I/O path compared with mmap. But in real Lucene workloads where I/O and computation interleave, mmap's async prefetch should compensate for some of that loss through overlap, the gap should be narrowed or eliminated when there's real work between prefetch and access. Again, this is purely theoretical analysis from micro-benchmarking perspective. It depends on the real use case. As @gf2121 pointed out in the original issue, it'd be better to provide a specific workload (e.g., multi-threaded level, HNSW search with a given graph size vs. available RAM) to quantitatively vet which choice is better and whether mmap with proper advice and prefetch can solve for that particular scenario, or pread can be justified. --- For sequential benchmark, I also changed the code, one thing I couldn't reproduce from your earlier numbers: `batched prefetch: ~1.4 GB/s (qd 128: ~2.7 GB/s)` and `sliding 2 MB prefetch window ≈ 2.5 GB/s`. Could you share the setup details? Mac model, file size, fio (with what params in command line) or JMH or custom program? <details> <summary>Appendix a. Mac (M3 Pro) JMH results</summary> # MAC Macbook M3 Pro, 36G RAM, 26G free available RAM ## CASE 1. warm read File fit into memory completely, no real I/O. ``` sudo purge cat /Users/zhangxuv/workspace/data/bench_data_16G.dat > /dev/null java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/Users/zhangxuv/workspace/data/bench_data_16G.dat -Dbench.fileSizeMB=16384" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 38.124 ± 2.351 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 95.258 ± 0.441 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 75.621 ± 0.679 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 67.261 ± 0.545 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 37.494 ± 0.578 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 93.797 ± 1.933 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 72.778 ± 1.267 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 68.292 ± 0.450 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 61.463 ± 0.475 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 182.212 ± 1.028 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 204.441 ± 28.288 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 193.523 ± 4.335 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 61.272 ± 0.473 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 181.592 ± 1.649 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 197.461 ± 1.523 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 194.241 ± 4.506 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 78.187 ± 0.081 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 224.790 ± 1.192 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 284.791 ± 1.487 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 294.529 ± 5.742 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 77.870 ± 1.505 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 224.781 ± 1.189 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 285.160 ± 2.533 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 295.452 ± 2.310 ops/ms ``` ## CASE 2. memory constraints File barely fit into memory, a bit more than the total RAM size. ``` sudo purge cat /Users/zhangxuv/workspace/data/bench_data_32G.dat > /dev/null java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/Users/zhangxuv/workspace/data/bench_data_32G.dat -Dbench.fileSizeMB=32768" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 2.561 ± 0.568 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 8.210 ± 0.267 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 12.844 ± 0.633 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 17.591 ± 1.040 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 2.029 ± 0.075 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 7.580 ± 0.390 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 11.998 ± 0.953 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 14.713 ± 9.033 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 7.223 ± 5.674 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 23.008 ± 1.276 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 34.684 ± 6.812 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 47.601 ± 1.989 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 8.067 ± 0.947 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 21.756 ± 1.514 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 32.931 ± 3.974 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 42.449 ± 4.459 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 5.267 ± 0.454 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 17.899 ± 2.129 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 29.034 ± 2.606 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 44.071 ± 7.863 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 5.849 ± 1.252 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 16.615 ± 5.282 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 23.787 ± 5.305 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 32.736 ± 1.748 ops/ms ``` ## CASE 3. many cold reads File can fit into memory, total RAM can only hold half of the size of the file. No page cache clearing before each iteration. ``` sudo purge java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/Users/zhangxuv/workspace/data/bench_data_64G.dat -Dbench.fileSizeMB=65536" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 0.528 ± 0.146 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 2.528 ± 0.805 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 4.300 ± 0.048 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 5.798 ± 0.022 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 0.775 ± 0.019 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 2.652 ± 0.094 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 4.206 ± 0.063 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 5.741 ± 0.040 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 1.189 ± 0.057 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 3.319 ± 0.388 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 4.377 ± 0.563 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 5.028 ± 0.962 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 1.141 ± 0.018 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 3.254 ± 0.309 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 4.374 ± 0.556 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 5.015 ± 0.995 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 0.440 ± 0.032 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 1.984 ± 0.055 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 3.288 ± 0.342 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 4.444 ± 0.550 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 0.464 ± 0.010 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 1.536 ± 0.038 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 2.697 ± 0.214 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 3.391 ± 0.366 ops/ms ``` ## CASE 4. all cold reads File can fit into memory, total RAM can only hold half of the size of the file. Clear page cache before each iteration ``` sudo purge java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/Users/zhangxuv/workspace/data/bench_data_64G.dat -Dbench.fileSizeMB=65536 -Dbench.dropCaches=true" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 0.448 ± 0.046 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 1.590 ± 0.095 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 2.713 ± 0.025 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 3.633 ± 0.055 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 0.395 ± 0.019 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 1.612 ± 0.019 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 2.716 ± 0.037 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 3.653 ± 0.014 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 0.416 ± 0.058 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 1.639 ± 0.043 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 2.777 ± 0.010 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 3.693 ± 0.037 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 0.426 ± 0.050 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 1.640 ± 0.017 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 2.783 ± 0.051 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 3.639 ± 0.212 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 0.196 ± 0.007 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 0.984 ± 0.051 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 1.947 ± 0.089 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 3.054 ± 0.225 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 0.316 ± 0.016 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 0.699 ± 0.020 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 2.032 ± 0.039 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 2.444 ± 0.084 ops/ms ``` </details> <details> <summary>Appendix b. Linux EBS (EC2 c5.4xlarge) JMH results </summary> # Linux (EBS, io2, 20000 IOPS provisioned) c5.4xlarge, 16 vCPU, 32G RAM ## CASE 1. warm read ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches cat /home/ec2-user/environment/data/bench-16G.dat > /dev/null java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/home/ec2-user/environment/data/bench-16G.dat -Dbench.fileSizeMB=16384" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 24.793 ± 0.672 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 88.617 ± 1.440 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 153.229 ± 4.362 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 204.380 ± 7.314 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 23.962 ± 0.448 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 76.683 ± 1.070 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 93.341 ± 18.655 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 91.340 ± 2.926 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 29.795 ± 1.295 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 111.310 ± 1.632 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 209.018 ± 2.382 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 249.923 ± 5.756 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 29.550 ± 1.002 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 112.409 ± 1.273 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 204.856 ± 4.393 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 246.014 ± 18.995 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 38.950 ± 1.084 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 145.524 ± 6.223 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 267.597 ± 7.051 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 309.087 ± 3.963 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 39.555 ± 0.988 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 145.546 ± 2.958 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 266.681 ± 8.701 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 309.043 ± 5.350 ops/ms ``` ## CASE 2. memory constraints ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches cat /home/ec2-user/environment/data/bench-32G.dat > /dev/null java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/home/ec2-user/environment/data/bench-32G.dat -Dbench.fileSizeMB=32768" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 1.367 ± 0.255 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 5.290 ± 0.614 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 5.873 ± 0.461 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 5.474 ± 0.143 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 1.203 ± 0.040 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 4.927 ± 0.207 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 5.270 ± 0.113 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 5.093 ± 0.294 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 0.961 ± 0.068 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 3.912 ± 0.125 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 4.746 ± 1.233 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 4.544 ± 1.225 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 0.536 ± 0.014 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 2.156 ± 0.086 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 3.646 ± 0.076 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 3.612 ± 0.074 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 0.535 ± 0.028 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 2.206 ± 0.045 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 3.670 ± 0.109 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 3.629 ± 0.055 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 0.912 ± 0.099 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 3.593 ± 0.613 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 3.775 ± 0.499 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 4.005 ± 0.623 ops/ms ``` ## CASE 3. many cold reads ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/home/ec2-user/environment/data/bench-64G.dat -Dbench.fileSizeMB=65536" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 0.240 ± 0.007 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 0.917 ± 0.081 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 1.704 ± 0.199 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 1.541 ± 0.075 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 0.211 ± 0.004 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 0.837 ± 0.038 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 1.434 ± 0.034 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 1.398 ± 0.026 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 0.132 ± 0.002 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 0.512 ± 0.036 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 0.520 ± 0.013 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 0.524 ± 0.011 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 0.071 ± 0.003 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 0.279 ± 0.015 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 0.453 ± 0.012 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 0.465 ± 0.002 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 0.072 ± 0.001 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 0.286 ± 0.008 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 0.462 ± 0.008 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 0.464 ± 0.007 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 0.134 ± 0.002 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 0.525 ± 0.017 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 0.512 ± 0.011 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 0.519 ± 0.012 ops/ms ``` ## CASE 4. all cold reads ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/home/ec2-user/environment/data/bench-64G.dat -Dbench.fileSizeMB=65536 -Dbench.dropCaches=true" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 0.130 ± 0.004 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 0.527 ± 0.014 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 1.075 ± 0.023 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 1.358 ± 0.045 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 0.131 ± 0.003 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 0.536 ± 0.016 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 1.077 ± 0.041 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 1.358 ± 0.046 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 0.071 ± 0.002 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 0.304 ± 0.015 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 0.320 ± 0.027 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 0.323 ± 0.029 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 0.038 ± 0.001 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 0.154 ± 0.003 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 0.262 ± 0.007 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 0.261 ± 0.007 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 0.038 ± 0.001 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 0.156 ± 0.005 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 0.262 ± 0.006 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 0.261 ± 0.007 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 0.072 ± 0.001 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 0.309 ± 0.021 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 0.319 ± 0.027 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 0.324 ± 0.032 ops/ms ``` </details> <details> <summary>Appendix c. Linux NVME (EC2 c6id.4xlarge) JMH results </summary> # Linux (NVME ssd) c6id.4xlarge, 16 vCPU, 32G RAM ## CASE 1. warm read ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches cat /mnt/local/bench-16G.dat > /dev/null java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-16G.dat -Dbench.fileSizeMB=16384" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 32.095 ± 0.261 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 114.121 ± 2.340 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 209.150 ± 1.613 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 266.454 ± 0.555 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 30.458 ± 1.220 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 89.328 ± 4.642 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 92.232 ± 6.642 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 97.084 ± 0.835 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 42.240 ± 0.943 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 159.396 ± 2.973 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 293.954 ± 1.888 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 325.528 ± 0.756 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 42.516 ± 0.277 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 159.545 ± 0.713 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 294.328 ± 1.730 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 324.951 ± 2.168 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 46.221 ± 2.781 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 176.713 ± 0.242 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 322.858 ± 0.969 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 336.820 ± 0.320 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 47.141 ± 0.064 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 176.865 ± 4.439 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 322.449 ± 2.347 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 336.823 ± 0.191 ops/ms ``` ## CASE 2. memory constraints ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches cat /mnt/local/bench-32G.dat > /dev/null java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-32G.dat -Dbench.fileSizeMB=32768" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 4.140 ± 0.127 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 15.387 ± 0.316 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 26.904 ± 1.388 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 33.164 ± 0.680 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 3.706 ± 0.056 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 13.836 ± 0.164 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 24.745 ± 0.305 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 33.025 ± 0.552 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 3.220 ± 0.139 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 11.208 ± 0.414 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 16.023 ± 1.615 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 16.440 ± 1.746 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 2.161 ± 0.119 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 8.326 ± 0.080 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 15.053 ± 0.195 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 25.252 ± 0.529 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 2.123 ± 0.082 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 8.329 ± 0.250 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 15.002 ± 0.452 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 25.368 ± 0.847 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 2.846 ± 0.361 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 10.548 ± 0.344 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 18.013 ± 0.744 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 23.574 ± 8.334 ops/ms ``` ## CASE 3. many cold reads ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 0.540 ± 0.009 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 2.080 ± 0.032 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 3.925 ± 0.027 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 5.228 ± 0.123 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 0.529 ± 0.003 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 2.037 ± 0.008 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 3.849 ± 0.015 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 5.028 ± 0.036 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 0.455 ± 0.008 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 1.209 ± 0.042 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 1.217 ± 0.028 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 1.226 ± 0.041 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 0.276 ± 0.005 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 1.066 ± 0.015 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 1.978 ± 0.017 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 3.370 ± 0.026 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 0.277 ± 0.004 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 1.068 ± 0.005 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 1.983 ± 0.011 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 3.366 ± 0.017 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 0.446 ± 0.015 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 1.210 ± 0.028 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 1.221 ± 0.029 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 1.223 ± 0.026 ops/ms ``` ## CASE 4. all cold reads ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar RandomReadIOBenchmark \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536 -Dbench.dropCaches=true" \ -p readSize=16384 -p readsPerOp=16 ``` ``` Benchmark (readSize) (readsPerOp) Mode Cnt Score Error Units RandomReadIOBenchmark.ffiPread_T01 16384 16 thrpt 6 0.293 ± 0.001 ops/ms RandomReadIOBenchmark.ffiPread_T04 16384 16 thrpt 6 1.157 ± 0.004 ops/ms RandomReadIOBenchmark.ffiPread_T08 16384 16 thrpt 6 2.272 ± 0.004 ops/ms RandomReadIOBenchmark.ffiPread_T16 16384 16 thrpt 6 4.383 ± 0.005 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T01 16384 16 thrpt 6 0.292 ± 0.001 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T04 16384 16 thrpt 6 1.155 ± 0.006 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T08 16384 16 thrpt 6 2.270 ± 0.010 ops/ms RandomReadIOBenchmark.fileChannelDirectBuffer_T16 16384 16 thrpt 6 4.376 ± 0.005 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T01 16384 16 thrpt 6 0.270 ± 0.017 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T04 16384 16 thrpt 6 0.863 ± 0.263 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T08 16384 16 thrpt 6 0.872 ± 0.265 ops/ms RandomReadIOBenchmark.mmapBatchedPrefetch_T16 16384 16 thrpt 6 0.940 ± 0.321 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T01 16384 16 thrpt 6 0.152 ± 0.001 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T04 16384 16 thrpt 6 0.602 ± 0.020 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T08 16384 16 thrpt 6 1.155 ± 0.078 ops/ms RandomReadIOBenchmark.mmapMadvRandomBatchedPrefetch_T16 16384 16 thrpt 6 2.080 ± 0.256 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T01 16384 16 thrpt 6 0.152 ± 0.001 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T04 16384 16 thrpt 6 0.603 ± 0.023 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T08 16384 16 thrpt 6 1.155 ± 0.082 ops/ms RandomReadIOBenchmark.mmapMadvRandom_T16 16384 16 thrpt 6 2.081 ± 0.256 ops/ms RandomReadIOBenchmark.mmap_T01 16384 16 thrpt 6 0.270 ± 0.017 ops/ms RandomReadIOBenchmark.mmap_T04 16384 16 thrpt 6 0.866 ± 0.261 ops/ms RandomReadIOBenchmark.mmap_T08 16384 16 thrpt 6 0.871 ± 0.271 ops/ms RandomReadIOBenchmark.mmap_T16 16384 16 thrpt 6 0.939 ± 0.323 ops/ms ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
