neoremind commented on PR #16279: URL: https://github.com/apache/lucene/pull/16279#issuecomment-4913556772
Thank you @mikemccand! I've updated the benchmark results and analysis above, please see my comments inline below. > I love that we are building tooling for benchmarking cold/warm cases -- they are so tricky to properly test because at a "whole search system" level you should replay query traffic with accurate actual production arrival times (ideally, or simulate w/ Poisson process) for real measures. I'd love to see Lucene eventually be able to saturate modern NVMe SSDs when executing a single query on all CPU cores on a larger-than-RAM index but I don't think we are there yet (we don't have enough async/concurrent/prefetch IO? and we don't strongly separate dependent IO into their own paths so non-dependent IOPs don't block one another). I fully agree on the vision. I think it's genuinely possible to saturate I/O as long as we interleave computation and I/O with parallelization. I've practiced this strategy before, during the 1st PolarDB database performance competition in 2018, I implemented a KV engine saturating an Intel Optane SSD with 2.2 GB/s write and 2.5 GB/s random read (see [write-up](https://github.com/neoremind/2018-polar-race/blob/master/building-a-kv-engine-to-saturate-optane-ssd-java.md)). I also built a [fast InnoDB checksum tool](https://github.com/neoremind/fast_innochecksum) in C++ that fully exploits hardware by separating I/O from computation, using big-block I/O to amortize kernel overhead, and making full use of multi-core. But indeed, these are either contest simplified workloads or easy utilities, Lucene's query path is much much more complex. One thing I learned from @jimczi is how `MMapDirectory`'s prefetch is already working good now in `StoredFields#prefetch(docID)` prefetching, and `K nnVectorValues#prefetch(int[] ords, int numOrds)` in `PrefetchableFlatVectorScorer` does the same for vector search, and more. > I also want to test if Lucene is anywhere near the linux kernel bottleneck limit on page faults / sec. This is a potential risk we face with memory-mapped IO (described here, but I think there's been good progress in very modern kernels, something about maple trees -- aha, here!). I think this benchmark should be able to tease out that bottleneck by comparing mmap IO vs NIO cold concurrent IOPs. Thanks for sharing these. The per-VMA lock optimization should be available after Linux 6.4, but my EC2 hosts are still running old kernels. I will give it try once available. With the data shows above, mmap is the most powerful one when data is hot in RAM, but pread/NIOFS outperforms plain mmap under memory pressure with some degree of cold load (case 2: file size nearly RAM, case 3: 50% cold reads, case 4: almost all cold). On NVME at T16 in the all-cold case: plain mmap can only do 0.94 ops/ms while pread reaches 4.65 ops/ms, however, mmap (RANDOM) + batched WILLNEED prefetch (simulate QD=16) goes to 6.64 ops/ms, beating both pread and NIOFS. This somehow shows that I could hit the mmap bottleneck (page table contention, TLB shootdowns, mmap lock contention) on the cold path, and batched prefetch makes mmap more performant by pipelining I/O and computation. > How do you simulate the memory pressure? I see you drop OS's IO cache (at the start of each run?). Is it just that the user is expected to test on a large enough file exceeding their free RAM. You could also use @rmuir's awesome ramhog.c in luceneutil: https://github.com/mikemccand/luceneutil/blob/main/src/python/ramhog.c ... it just allocates and pins that amount of RAM so OS cannot use it anymore. I use files that go beyond available RAM, 32G/64G file on a 32G host. Just for the "almost all cold" case 4, I drop page cache before each JMH iteration, others no, just let program contended on memory. The dropping page cache is a switch as param. I also tried using cgroup to limit memory, but ran into a tricky issue, dropping page cache within the cgroup doesn't fully remove pages from host memory, anyway, I didn't go deeper the why. So, I ended up using a host with less RAM instead. I think @rmuir's `ramhog.c` would be a cleaner approach for memory control, I will take a look. > Have you tried comparing this benchy to other IO benchy tools e.g. fio? Then we could gain some confidence in our benchy... (or, maybe lose some confidence in fio!). Yes, please see the fio section in the results above. In short: JMH numbers match fio with tiny overhead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
