rmuir commented on PR #11917: URL: https://github.com/apache/lucene/pull/11917#issuecomment-1313991795
> @uschindler I think I understand Robert's concerns against mlock, but less yours against loading. Do you think that it would be useless to preload if these files can then get paged out? My thinking was that most users would rather like Lucene to take a bit longer to open indices if it helps provide better performance on initial queries, noting that it wouldn't make opening much slower given how tiny these files tend to be compared to the overall dataset (~0.17% of the total index size for the index generated by nightly benchmarks). In general these files would stay in the page cache all the time, but if they don't because of unusual access patterns or competing processes and the page cache decides to page out some data, it wouldn't be much worse than today. I do think some users will see appeal in the mlock option and getting stronger guarantees that this data wouldn't get paged out, but this feels like something that should be an opt-in rather than the default? I'm not against preloading i just don't think it does anything here. It just madvises the kernel for read-ahead and then touches every page. Seems something went badly wrong here: if these files need to be hot, then just load the structures into the heap? all this talk about preloading and locking pages is not good. why were these "small hot files" removed from the heap? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
