jimczi commented on issue #16767: URL: https://github.com/apache/lucene/issues/16767#issuecomment-5966898672
I opened #16769 to handle the missing formats. One thing to flag: RANDOM probably isn’t the right advice for vectors used within HNSW. Setting RANDOM on a mapping means accesses don’t update recency, so once the page cache is full, reclamation starts with those pages. As a result, the HNSW graph and its associated vectors (raw or quantized) are the first to be reclaimed when memory runs low. That’s also why we added the NO_REUSE hint for raw vectors used during rescoring. Raw vectors are generally the largest files in the index and are often bigger than the available page cache. Using RANDOM there makes sense because it keeps the other hot files hot while the vectors use only a small page-cache buffer for rescoring. In our tests, this approach is even competitive with direct I/O, without requiring dedicated threads for the reads, since we can take advantage of prefetching. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
