rmuir commented on PR #13998: URL: https://github.com/apache/lucene/pull/13998#issuecomment-2483324513
> Yeah, we can look at how to call `mincore`, and it might not be that much of a lift with the existing plumbing. Maybe something can look at as a follow up? I'm really trying to get to a situation where we can load (`MADV_WILLNEED`), and check even the HNSW graph. Maybe even `mlock`, as a potential follow up. Since not having the graph in memory results in horrible perf (need to get some numbers). yes, agreed about `mincore` as a followup. Let's use existing JDK plumbing as a start as done here. i'm very much against using `mlock`, there are so many problems with this. With an out of box linux system my ulimit for this is set to 8MB. I really don't think we should be mlocking gigabytes of vectors because the access is inefficient. It would be better to improve documentation, so that users avoid the typical mistakes such as setting too-big java heap (leaving no room for buffers/cache), configure swappiness if needed, etc. mlock will just make problems worse. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
