mikemccand opened a new issue, #16767: URL: https://github.com/apache/lucene/issues/16767
### Description [Spinoff from [this comment](https://github.com/apache/lucene/pull/15979#discussion_r4168808020)] Not sure if this is a silly idea or might be helpful: We have painful cases today where the same mapped file sometimes should be optimized with `MADV_RANDOM`, say when reading full precision vectors for re-ranking in a `VectorQuery`, or sometimes with `MADV_SEQUENTIAL`, say when merging. The hint is surprisingly impactful! I've seen benchmarks take many minutes (when it should've been a few seconds) on a fast-ish NVMe SSD because the vector files were hinted the wrong way. We have fragile logic that will flip the hint of the same mapped region to `SEQUENTIAL` when a merge is running, then back to `RANDOM` when the merge is done. But couldn't we just map the same file twice, so we don't have to toggle the hinting? The kernel will still load the pages once from file and share them across both (or more than two) maps. Claude Opus 5.5 Medium [seems to think modern Linux kernels would do the right thing](https://claude.ai/share/e3c1b5ea-a22e-42b1-ab26-44fdf22c91eb) as long as we also `open(...)` the file twice (two file descriptors) -- i.e. the hints would stay separate. But maybe it's optimistically hallucinating for me... There would be an added cost of minor page faults for the kernel to share a single cached page in the buffer cache for both maps. I.e. if a page became hot because of readahead from the merging, when a query then needs that page, there is a minor page fault to do the accounting so that the `RANDOM` map shares that same hot page. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
