mikemccand commented on PR #16279: URL: https://github.com/apache/lucene/pull/16279#issuecomment-4903092429
I love that we are building tooling for benchmarking cold/warm cases -- they are so tricky to properly test because at a "whole search system" level you should replay query traffic with accurate actual production arrival times (ideally, or simulate w/ [Poisson process](https://en.wikipedia.org/wiki/Poisson_point_process)) for real measures. I'd love to see Lucene eventually be able to saturate modern NVMe SSDs when executing a single query on all CPU cores on a larger-than-RAM index but I don't think we are there yet (we don't have enough async/concurrent/prefetch IO? and we don't strongly separate dependent IO into their own paths so non-dependent IOPs don't block one another). I also want to test if Lucene is anywhere near the linux kernel bottleneck limit on page faults / sec. This is a potential risk we face with memory-mapped IO ([described here](https://lwn.net/Articles/906852/), but I think there's been good progress in very modern kernels, [something about maple trees -- aha, here](https://kernel-internals.org/mm/per-vma-locks/)!). I think this benchmark should be able to tease out that bottleneck by comparing mmap IO vs NIO cold concurrent IOPs. How do you simulate the memory pressure? I see you drop OS's IO cache (at the start of each run?). Is it just that the user is expected to test on a large enough file exceeding their free RAM. You could also use @rmuir's awesome `ramhog.c` in luceneutil: https://github.com/mikemccand/luceneutil/blob/main/src/python/ramhog.c ... it just allocates ans pins that amount of RAM so OS cannot use it anymore. Have you tried comparing this benchy to other IO benchy tools e.g. [`fio`](https://fio.readthedocs.io/en/latest/fio_doc.html)? Then we could gain some confidence in our benchy... (or, maybe lose some confidence in `fio`!). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
