epotyom opened a new issue, #15515: URL: https://github.com/apache/lucene/issues/15515
In https://github.com/apache/lucene/pull/15458 we found that cold index optimizations (deferring term states to send the madvise WILLNEED signal) can hurt performance when running a hot index. This might also be true for other cases where Lucene uses madvise? We could have a parameter for users to select hot or cold index mode, depending on their use case. We can also try to implement an adaptive mode, which enables cold index optimizations when the index is cold and enables hot index optimizations otherwise. I thought (https://github.com/apache/lucene/pull/15458#issuecomment-3617935050) we could use the `isLoaded` method for it, but as @mikemccand pointed out, this method is quite expensive (https://github.com/apache/lucene/pull/14156#issuecomment-3649593085). I think we can implement an `isProbablyLoaded` method (which checks only the first, middle, and last pages of a memory segment) to overcome that. As for `TermStates.get` itself, I think we can propagate the `isProbablyLoaded` method results to it and defer the dictionary lookup only if the index is cold. We can also look at other madvise use cases and see if we can apply the same mechanism to them? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
