neoremind commented on PR #16279: URL: https://github.com/apache/lucene/pull/16279#issuecomment-4913504832
I ran the benchmark across three different hardware profiles to get quantitative measurements of how different I/O strategies work under different memory pressure. Here are the results. ## Platforms | Platform | CPU | RAM | Storage | 4K random read latency (QD=1) | Saturated random read throughput | |---|---|---|---|---|---| | Mac (M3 Pro) | 12 CPU | 36G | 512G Apple SSD | ~77us | ~840 MB/s | | Linux EBS (EC2 c5.4xlarge) | 16 vCPU | 32G | EBS io2, 200G, 20K provisioned IOPS | ~334us | ~80 MB/s | | Linux NVME (EC2 c6id.4xlarge) | 16 vCPU | 32G | NVME SSD | ~103us | ~1.25 GB/s | *The latency and throughput numbers are calculated via `fio` with rand read 16k + different iodepth/numjobs to find the saturation point.* ## Benchmark Setup 16k reads (4 pages) x 16 reads/op = 256KB per op. Throughput (MB/s) = ops/ms × 256k x 1000. For example, with O_DIRECT on c6id.4xlarge (Linux NVME SSD), JMH at 16 threads yields 4.9 ops/ms → `4.9 × 256k x 1000 ≈ 1250 MB/s`, matching the peak random read throughput from `fio`. I removed the O_DIRECT results from the table charts below to focus on mmap (with `MMapDirectory`), pread (via FFI), and `NIOFSDirectory` (backed by FileChannel). ## Random Read Results ### CASE 1: Fully warm (all data in page cache) 16G file fits in RAM, pre-warmed with `cat file > /dev/null`. <img width="672" height="378" alt="warm-mac" src="https://github.com/user-attachments/assets/6aef0f7f-65e5-47a6-b7b0-47544783ef72"; /> <img width="662" height="395" alt="warm-linux-ebs" src="https://github.com/user-attachments/assets/f4e25b59-3913-40d2-8116-6c99fbf92937"; /> <img width="636" height="387" alt="warm-linux-nvme" src="https://github.com/user-attachments/assets/b84a59a6-4f81-49f7-8345-399fd21413e6"; /> - When everything is in RAM, mmap removes syscall overhead, it's pure page-table lookup with direct pointer to cached page, also no intermediate offheap buffer copy in NIO FileChannel. At T8, mmap is 1.5-2x faster than pread and nio FileChannel. - Observed that batched WILLNEED prefetch adds ~25–30% overhead since the `posix_madvise` syscall is not necessary when pages are already in RAM (see [pure mmap w/o using `MMapDirectory` in Lucene](https://github.com/apache/lucene/issues/16044#issuecomment-4830551502)). But here `MemorySegmentIndexInput#prefetch` uses a power-of-two backoff counter to skip prefetch when pages are loaded, this avoids the slightly `posix_madvise` penalty for hot data and high IOPS scenarios, so we can see there is no difference in performance between all variants of mmap. - `NIOFSDirectory` (backed by FileChannel) hits a thread-scaling wall, past ~4 threads, the `NativeThreadSet` monitor contention bottlenecked performance as found in #16044. Pread doesn't suffer from this. <details> <summary>Detailed JMH results (ops/ms, higher is better)</summary> #### Mac M3 Pro (36G RAM, unified memory, Apple SSD) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 77.98 | 224.69 | 285.09 | 296.12 | | mmap (RANDOM) | 77.99 | 224.72 | 285.42 | 295.74 | | mmap + batchedPrefetch | 77.62 | 223.26 | 284.43 | 294.45 | | mmap (RANDOM) + batchedPrefetch | 77.78 | 222.51 | 284.75 | 293.86 | | ffiPread | 37.43 | 93.84 | 73.71 | 65.64 | | fileChannelNIOFS | 28.05 | 80.32 | 67.77 | 66.43 | #### Linux c5.4xlarge (16 vCPU, 32G RAM, EBS io2 20K IOPS) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 37.46 | 141.00 | 261.55 | 282.78 | | mmap (RANDOM) | 36.93 | 141.52 | 261.13 | 282.63 | | mmap + batchedPrefetch | 37.00 | 138.04 | 257.00 | 280.10 | | mmap (RANDOM) + batchedPrefetch | 36.54 | 138.23 | 256.77 | 280.09 | | ffiPread | 22.38 | 77.67 | 134.53 | 185.46 | | fileChannelNIOFS | 18.38 | 58.24 | 82.17 | 77.16 | | ffiPreadDirectIO (O_DIRECT) | 0.15 | 0.63 | 1.27 | 1.25 | #### Linux c6id.4xlarge (16 vCPU, 32G RAM, local NVME SSD) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 45.16 | 174.57 | 316.09 | 332.71 | | mmap (RANDOM) | 46.53 | 173.58 | 316.22 | 332.91 | | mmap + batchedPrefetch | 46.22 | 171.98 | 312.48 | 330.43 | | mmap (RANDOM) + batchedPrefetch | 45.25 | 171.93 | 312.14 | 330.34 | | ffiPread | 31.53 | 114.05 | 204.93 | 261.52 | | fileChannelNIOFS | 24.57 | 78.74 | 102.25 | 97.01 | | ffiPreadDirectIO (O_DIRECT) | 0.66 | 2.47 | 4.46 | 4.80 | </details> ### CASE 2: Memory pressure (working set close to RAM) 32G file barely exceeds RAM, pre-warmed. <img width="673" height="380" alt="mem-pressure-mac" src="https://github.com/user-attachments/assets/b4a3f0ab-9a40-4afe-baa5-90a034651d0f"; /> <img width="665" height="401" alt="mem-pressure-linux-ebs" src="https://github.com/user-attachments/assets/5f7c27c5-5b65-48bb-83c9-77b7a533daac"; /> <img width="637" height="386" alt="mem-pressure-linux-nvme" src="https://github.com/user-attachments/assets/1a3aa38d-e7fe-488b-8d6e-257b9a16377b"; /> - On Mac, mmap is the best strategy. - `NIOFSDirectory` thread contention disappears when there are quite amount of I/O, since the I/O latency dwarfs the monitor overhead, so threads are no longer hot-contending on `NativeThreadSet`. - On Linux, pread beats plain mmap at all thread counts with memory pressure not matter how big. I think mmap's page-fault path is more expensive than pread's syscall on Linux, especially under memory pressure. This paper from Andy Pavlo ["Are You Sure You Want to Use MMAP in Your Database Management System?"](https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf) argues mmap is not the best option for larger-than-memory DBMS workloads if not used carefully. It notes InfluxDB, MongoDB, and SingleStore deprecated mmap in their latest releases. Section 3.4 (Problem-4: Performance Issues) in the paper points to OS page eviction mechanisms that don't scale beyond a few threads for larger-than-memory workloads on high-bandwidth storage with bottlenecks like page table contention, page eviction overhead, and TLB shootdowns. @mikemccand also pointed out the mmap lock bottleneck above. There's also a [counter-argument](https://www.scylladb.com/2022/05/19/mmap-is-not-the-best-option-fo r-dbms-is-it/) worth reading for balance. - This is a case where we still hit hot pages most of the time, but whenever we encounter I/O, mmap is not as good as pread, and what's worse, the backoff mechanism weakens mmap + prefetch. Batched prefetch should theoretically be the most competitive here (as demonstrated in [#16044](https://github.com/apache/lucene/issues/16044#issuecomment-4830551502)'s benchmarks using customized mmap + prefetch without using `MMapDirectory`), but the power-of-two backoff counter causes it to skip most `madvise` calls, so its benefit diminishes or gets emptied away. All in all, without prefetch, mmap alone is not competitive with pread under memory pressure on Linux, prefetch is the silver bullet that makes mmap performant in this scenario. <details> <summary>Detailed JMH results (ops/ms, higher is better)</summary> #### Mac M3 Pro (36G RAM, unified memory, Apple SSD) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 4.76 | 15.73 | 23.50 | 34.65 | | mmap (RANDOM) | 4.80 | 15.02 | 24.42 | 33.40 | | mmap + batchedPrefetch | 2.54 | 19.52 | 28.09 | 38.22 | | mmap (RANDOM) + batchedPrefetch | 5.28 | 16.54 | 23.17 | 32.46 | | ffiPread | 2.20 | 7.60 | 12.10 | 16.94 | | fileChannelNIOFS | 1.87 | 6.86 | 11.11 | 16.02 | #### Linux c5.4xlarge (16 vCPU, 32G RAM, EBS io2 20K IOPS) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 0.85 | 3.46 | 3.62 | 3.82 | | mmap (RANDOM) | 0.50 | 2.03 | 3.38 | 3.35 | | mmap + batchedPrefetch | 0.93 | 3.81 | 4.50 | 4.33 | | mmap (RANDOM) + batchedPrefetch | 0.50 | 2.00 | 3.33 | 3.38 | | ffiPread | 1.48 | 5.15 | 5.48 | 5.12 | | fileChannelNIOFS | 1.12 | 4.40 | 4.84 | 4.78 | | ffiPreadDirectIO (O_DIRECT) | 0.15 | 0.63 | 1.28 | 1.25 | #### Linux c6id.4xlarge (16 vCPU, 32G RAM, local NVME SSD) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 3.29 | 11.51 | 18.86 | 24.14 | | mmap (RANDOM) | 2.14 | 8.45 | 15.29 | 26.36 | | mmap + batchedPrefetch | 3.96 | 13.21 | 16.49 | 20.07 | | mmap (RANDOM) + batchedPrefetch | 2.25 | 8.58 | 15.52 | 25.87 | | ffiPread | 6.04 | 19.05 | 31.47 | 33.87 | | fileChannelNIOFS | 4.19 | 15.32 | 26.14 | 29.54 | | ffiPreadDirectIO (O_DIRECT) | 0.66 | 2.46 | 4.45 | 4.80 | </details> ### CASE 3: Many cold reads (~50% cold reads) 64G file (2x available RAM), ~50% cold reads, pre-warmed. <img width="672" height="381" alt="half-cold-read-mac" src="https://github.com/user-attachments/assets/bc612e43-2f86-4347-9c77-256c46364373"; /> <img width="665" height="400" alt="half-cold-read-linux-ebs" src="https://github.com/user-attachments/assets/a9191135-0c86-4cce-8f9b-1415432bafd1"; /> <img width="637" height="384" alt="half-cold-read-linux-nvme" src="https://github.com/user-attachments/assets/9c33dff0-9102-4467-b9b0-d3cf2b65824a"; /> - mmap + batched prefetch catches up or surpasses pread here. With ~50% of pages cold, there's a much higher chance for willneed prefetch to actually trigger async I/O operations that deepen iodepth and fill the I/O queue. Even at T01, the prefetch benefit is already very obvious, throughput climbs up high with just one thread due to batched async I/O. - Interestingly, RANDOM is particularly useful under high-concurrency on fast NVME, likely because it avoids wasted readahead pages that would otherwise be evicted before use. On EBS however, NORMAL wins, the kernel's sequential readahead amortizes the high per-request latency of remote block fetches. <details> <summary>Detailed JMH results (ops/ms, higher is better)</summary> #### Mac M3 Pro (36G RAM, unified memory, Apple SSD) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 0.45 | 1.55 | 2.58 | 3.27 | | mmap (RANDOM) | 0.45 | 1.97 | 3.17 | 4.28 | | mmap + batchedPrefetch | 1.07 | 2.72 | 3.66 | 4.32 | | mmap (RANDOM) + batchedPrefetch | 0.98 | 2.83 | 3.84 | 4.61 | | ffiPread | 0.58 | 2.53 | 4.16 | 5.53 | | fileChannelNIOFS | 0.78 | 2.64 | 4.02 | 5.44 | #### Linux c5.4xlarge (16 vCPU, 32G RAM, EBS io2 20K IOPS) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 0.14 | 0.49 | 0.49 | 0.52 | | mmap (RANDOM) | 0.07 | 0.28 | 0.46 | 0.46 | | mmap + batchedPrefetch | 0.89 | 1.51 | 1.52 | 1.50 | | mmap (RANDOM) + batchedPrefetch | 0.66 | 1.50 | 1.49 | 1.48 | | ffiPread | 0.25 | 0.96 | 1.67 | 1.52 | | fileChannelNIOFS | 0.22 | 0.88 | 1.42 | 1.38 | | ffiPreadDirectIO (O_DIRECT) | 0.15 | 0.62 | 1.27 | 1.25 | #### Linux c6id.4xlarge (16 vCPU, 32G RAM, local NVME SSD) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 0.64 | 1.21 | 1.22 | 1.23 | | mmap (RANDOM) | 0.28 | 1.09 | 2.01 | 3.41 | | mmap + batchedPrefetch | 3.63 | 5.75 | 6.14 | 6.26 | | mmap (RANDOM) + batchedPrefetch | 3.99 | 7.21 | 7.20 | 7.21 | | ffiPread | 1.06 | 3.59 | 5.39 | 5.15 | | fileChannelNIOFS | 0.88 | 3.23 | 5.15 | 4.98 | | ffiPreadDirectIO (O_DIRECT) | 0.66 | 2.47 | 4.45 | 4.80 | </details> ### CASE 4: Almost all cold reads 64G file (2x available RAM), clear page cache before each iteration, cold start. <img width="673" height="381" alt="all-cold-mac" src="https://github.com/user-attachments/assets/13e0718b-3360-4bf3-a81a-c169077c4f2c"; /> <img width="663" height="400" alt="all-cold-linux-ebs" src="https://github.com/user-attachments/assets/a053bc8f-8fd9-46a6-8513-2e6a2ec0f3e7"; /> <img width="639" height="389" alt="all-cold-linux-nvme" src="https://github.com/user-attachments/assets/c66bb3fe-b5da-4b79-b70a-17cce0fc8009"; /> - Same story as Case 3, but the gap widens since mmap + batchedPrefetch wins more on both EBS and NVME. - The current `MMapDirectory` + prefetch backoff strategy is well-designed for the fully warm and almost all code reads, but when data is mix of warm and cold, the backoff counter ramps up from hot-page hits and skips the batched `WILLNEED` hints diminishing the effect. So whether to enable prefetch all the time is not one-size-fits-all, it depends on query pattern like warm/cold read ratio and what if warm loads are accessed first. This aligns with my finding in https://github.com/apache/lucene/pull/16145#issuecomment-4594402925. <details> <summary>Detailed JMH results (ops/ms, higher is better)</summary> #### Mac M3 Pro (36G RAM, unified memory, Apple SSD) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 0.30 | 0.67 | 1.97 | 2.29 | | mmap (RANDOM) | 0.19 | 0.98 | 1.84 | 2.85 | | mmap + batchedPrefetch | 0.40 | 1.49 | 2.56 | 3.45 | | mmap (RANDOM) + batchedPrefetch | 0.42 | 1.51 | 2.54 | 3.38 | | ffiPread | 0.41 | 1.58 | 2.61 | 3.38 | | fileChannelNIOFS | 0.38 | 1.55 | 2.56 | 3.41 | | ffiPreadDirectIO (F_NOCACHE) | 0.44 | 1.97 | 3.56 | 5.68 | #### Linux c5.4xlarge (16 vCPU, 32G RAM, EBS io2 20K IOPS) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 0.08 | 0.33 | 0.32 | 0.33 | | mmap (RANDOM) | 0.04 | 0.16 | 0.26 | 0.26 | | mmap + batchedPrefetch | 1.36 | 1.35 | 1.35 | 1.38 | | mmap (RANDOM) + batchedPrefetch | 1.34 | 1.35 | 1.35 | 1.37 | | ffiPread | 0.14 | 0.57 | 1.15 | 1.36 | | fileChannelNIOFS | 0.14 | 0.58 | 1.15 | 1.36 | | ffiPreadDirectIO (O_DIRECT) | 0.16 | 0.63 | 1.27 | 1.28 | #### Linux c6id.4xlarge (16 vCPU, 32G RAM, local NVME SSD) | Benchmark | T01 | T04 | T08 | T16 | |-----------|-----|-----|-----|-----| | mmap (NORMAL) | 0.45 | 0.87 | 0.87 | 0.94 | | mmap (RANDOM) | 0.16 | 0.62 | 1.18 | 2.11 | | mmap + batchedPrefetch | 4.60 | 6.07 | 6.08 | 6.17 | | mmap (RANDOM) + batchedPrefetch | 4.53 | 6.26 | 6.26 | 6.64 | | ffiPread | 0.63 | 2.35 | 4.24 | 4.65 | | fileChannelNIOFS | 0.62 | 2.32 | 4.20 | 4.65 | | ffiPreadDirectIO (O_DIRECT) | 0.66 | 2.47 | 4.45 | 4.95 | </details> ## Sequential Read Results 64G file, page cache dropped before each iteration. 128 sequential reads per op at varying read sizes (16KB, 64KB, 128KB). Look at Linux, mmap (NORMAL) with default kernel readahead performs well across the board for sequential access. mmap (RANDOM) + batchedPrefetch is as good as mmap (NORMAL) with readahead, again, the power-of-two backoff diminishes the effect. <img width="533" height="279" alt="seq-mac" src="https://github.com/user-attachments/assets/4c2d5ca8-758e-42cb-83ac-e23a90701397"; /> <img width="532" height="278" alt="seq-linux-ebs" src="https://github.com/user-attachments/assets/45027548-2c07-4faf-99db-e6bf3a3ace9a"; /> <img width="533" height="277" alt="seq-linux-nvme" src="https://github.com/user-attachments/assets/d992edf8-7174-4ff2-9087-daa5a927fdc0"; /> <details> <summary>Detailed JMH results (ops/ms, higher is better)</summary> #### Mac M3 Pro (36G RAM, unified memory, Apple SSD) | Benchmark | 16KB | 64KB | 128KB | |-----------|------|------|-------| | mmap (NORMAL) | 4.032 | 0.993 | 0.503 | | mmap (SEQUENTIAL) | 2.292 | 0.601 | 0.288 | | mmap (RANDOM) | 1.839 | 0.452 | 0.231 | | mmap (RANDOM) + batchedPrefetch | 5.391 | 1.546 | 0.871 | | ffiPread | 2.009 | 0.661 | 0.577 | | fileChannelNIOFS | 2.123 | 0.510 | 0.261 | | ffiPreadDirectIO (F_NOCACHE) | 0.860 | 0.218 | 0.152 | #### Linux c5.4xlarge (16 vCPU, 32G RAM, EBS io2 20K IOPS) | Benchmark | 16KB | 64KB | 128KB | |-----------|------|------|-------| | mmap (NORMAL) | 2.223 | 0.601 | 0.272 | | mmap (SEQUENTIAL) | 2.236 | 0.563 | 0.281 | | mmap (RANDOM) | 0.101 | 0.024 | 0.011 | | mmap (RANDOM) + batchedPrefetch | 2.205 | 1.136 | 0.565 | | ffiPread | 1.922 | 0.553 | 0.275 | | fileChannelNIOFS | 2.276 | 0.491 | 0.266 | | ffiPreadDirectIO (O_DIRECT) | 0.163 | 0.072 | 0.035 | #### Linux c6id.4xlarge (16 vCPU, 32G RAM, local NVME SSD) | Benchmark | 16KB | 64KB | 128KB | |-----------|------|------|-------| | mmap (NORMAL) | 9.437 | 2.374 | 1.189 | | mmap (SEQUENTIAL) | 7.405 | 1.848 | 0.925 | | mmap (RANDOM) | 0.722 | 0.180 | 0.090 | | mmap (RANDOM) + batchedPrefetch | 9.816 | 2.454 | 1.224 | | ffiPread | 9.694 | 2.388 | 1.200 | | fileChannelNIOFS | 11.027 | 2.720 | 1.386 | | ffiPreadDirectIO (O_DIRECT) | 0.703 | 0.153 | 0.076 | </details> ## fio vs. JMH benchmarks I cross-validated against fio on c6id.4xlarge (NVME). The JMH numbers match fio within about 4% overhead (like JVM indirections, jmh blackhole?): - Sequential 1MB reads: JMH pread/NIOFS/mmap 12.01K IOPS vs. fio (sync/psync/libaio/mmap) 12.5K IOPS, both saturate disk bandwidth <details> <summary>Command used</summary> ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "SequentialReadIOBenchmark\.(fileChannelNIOFS)" \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \ -p readSize=1048576 -p readsPerOp=1 -r 30 sync && echo 3 | sudo tee /proc/sys/vm/drop_caches java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "SequentialReadIOBenchmark\.(ffiPread)" \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \ -p readSize=1048576 -p readsPerOp=1 -r 30 sync && echo 3 | sudo tee /proc/sys/vm/drop_caches java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "SequentialReadIOBenchmark\.(mmap)" \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \ -p readSize=1048576 -p readsPerOp=1 -r 30 fio --name=seqread \ --rw=read \ --bs=1m \ --size=64g \ --numjobs=1 \ --ioengine=psync \ --direct=1 \ --runtime=30 \ --time_based \ --group_reporting \ --filename=/mnt/local/bench-64G.dat fio --name=seqread \ --rw=read \ --bs=1m \ --size=64g \ --numjobs=1 \ --ioengine=mmap \ --direct=1 \ --runtime=30 \ --time_based \ --group_reporting \ --filename=/mnt/local/bench-64G.dat fio --name=seqread \ --rw=read \ --bs=1m \ --size=64g \ --numjobs=1 \ --ioengine=libaio \ --direct=1 \ --runtime=30 \ --time_based \ --group_reporting \ --filename=/mnt/local/bench-64G.dat ``` </details> - Random 16KB reads (4 threads, QD=1): JMH pread 40.5K IOPS vs. fio libaio 42.8K IOPS <details> <summary>Command used</summary> ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "RandomReadIOBenchmark\.(ffiPread_T04)" \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \ -p readSize=16384 -p readsPerOp=1 -f 1 -wi 1 -w 3 -i 1 -r 30 ## no direct set, so some page caches hit fio --name=randread \ --rw=randread \ --bs=16k \ --size=64g \ --iodepth=x \ --numjobs=4 \ --ioengine=libaio \ --runtime=30 \ --time_based \ --group_reporting \ --filename=/mnt/local/bench-64G.dat ``` </details> - Random 16KB with mmap: JMH and fio mmap (QD=1) can both achieve 12K IOPS <details> <summary>Command used</summary> ``` sync && echo 3 | sudo tee /proc/sys/vm/drop_caches java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "RandomReadIOBenchmark\.(mmap_T04)" \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \ -p readSize=16384 -p readsPerOp=1 -f 1 -wi 1 -w 3 -i 1 -r 10 fio --name=randread \ --rw=randread \ --bs=16k \ --size=64g \ --iodepth=1 \ --numjobs=4 \ --ioengine=mmap \ --runtime=30 \ --time_based \ --group_reporting \ --filename=/mnt/local/bench-64G.dat ``` </details> - Random 16KB with mmap: JMH (mmap + batched prefetch) and fio libaio (QD=16, numjobs=4) can both saturate I/O with 80K IOPS and 1.25G throughput. <details> <summary>Command used</summary> ``` # convert result by 5.0 ops/ms x 16 ops x 1000 = 80K IOPS # JMH can go even higher if we extend running time from 10 seconds to more because of more hot page cache hit sync && echo 3 | sudo tee /proc/sys/vm/drop_caches java -jar lucene/benchmark-jmh/build/benchmarks/lucene-benchmark-jmh-11.0.0-SNAPSHOT.jar "RandomReadIOBenchmark\.(mmapRandom_T04)" \ -jvmArgs "--enable-native-access=ALL-UNNAMED -Xms2g -Xmx2g -Dbench.file=/mnt/local/bench-64G.dat -Dbench.fileSizeMB=65536" \ -p readSize=16384 -p readsPerOp=16 -f 1 -wi 1 -w 1 -i 1 -r 10 fio --name=randread \ --rw=randread \ --bs=16k \ --size=64g \ --iodepth=16 \ --numjobs=4 \ --ioengine=libaio \ --direct=1 \ --runtime=30 \ --time_based \ --group_reporting \ --filename=/mnt/local/bench-64G.dat ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
