navneet1v opened a new issue, #13920: URL: https://github.com/apache/lucene/issues/13920
## Description For past 1 month we have been testing difference in performance for a files getting opened with IOContext.RANDOM vs IOContext.READ specially during merges with Lucene version 9.11.1 and Opensearch version 2.17. We started this deep-dive when we saw an increase in time for our force merges. ### Background Opensearch k-NN plugin provides the vector search capabilities with Opensearch. The architecture of k-NN plugin is very similar to how Lucene implements the vector search with few small key differences. Before 2.17 version of Opensearch, opensearch was using dvd files to store the raw the vectors and graphs files are stored separately. These graphs are not built using Lucene HNSW but with some external libraries like Faiss. In 2.17 version of Opensearch, we started using KNNFlatVectorFormat to store and read the raw vectors in place of dvd files. As reading vectors from .vec files as float[] is more efficient than reading byte[] and then converting to float[]. Ref: https://github.com/opensearch-project/k-NN/issues/1853 ## Observations After doing the switch what we observed that our merge time for a 10M 768D dataset is increased by [20%](https://github.com/opensearch-project/k-NN/issues/2134). We did an extensive deep-dive/experiments on the root cause([ref1](https://github.com/opensearch-project/k-NN/issues/2134#issuecomment-2415496614), [ref2](https://github.com/opensearch-project/k-NN/issues/2134#issuecomment-2415542711)) and code difference between dvd files and .vec files format and was able to see that IOContext.RANDOM with .vec files is causing this regression. This regression comes because during merges for every Lucene99FlatVectorsReader there are some operations happens like [checkIntegrity](https://github.com/apache/lucene/blob/branch_9_11/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsReader.java#L227-L229)(which does checksum of whole file), [reading of all vector values to create new segment](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java#L242-L266) which are more of sequential reads than random reads. I do believe that having a `RANDOM` madvise on `.vec` file is more beneficial for search and graph merges as Lucene uses this file as a way to store raw vectors for HNSW. BTW This PR added the capability: https://github.com/apache/lucene/pull/13267 for Random IOContext to .vec files. ## Solutions Tried: We have been trying multiple solution(all on Lucene version 9.11.1) and have been in touch with @uschindler and @mikemccand over java-u...@lucene.apache.org : 1. Switching the IOContext from RANDOM to READ: This actually helped in reducing the merges time so our original 20% increase in merge time was reduced to 0%. But making this change from RANDOM to READ for `.vec` added the extra latency for Lucene HNSW search. We saw 2x increase in latency because of this on a 10M 768D dataset. 2. Creating a new IndexInput in checkIntegrity function: This solution creates a new IndexInput in checkIntegrity function of Lucene99FlatVectorsReader with madvise as `SEQUENTIAL`. One of the biggest con it has is we are creating 2 indexinputs for same file in different threads and it not been recommended as per [this](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/IndexInput.java#L23-L40). 3. We also tried enabling the preload functionality for .vec file, it really helped in reducing the merge time similar to \#1, but suffered 2x latency increase during search. 4. We borrowed some code from latest version of Lucene of changing the mdvise before doing integrity checks and merging of flat vectors and applied it on 9.11.1 version. ref: https://github.com/shatejas/lucene/commit/4de387288d70b4d8aede45ef3095ae6c1e189331#diff-e0a29611df21f6d32a461e2d24db1585cdf3a8590f08d93b097f0dd84684ebc8R316, but with this we expect any search that is happening during the merges will have high latencies. This increase is a hypothesis we have not run the benchmarks around this. ### What is an ideal solution? In my mind an ideal solution will be the one which takes the advantage of different type of madvise and changes the madvise for the file based on the need(if merge of flatvectors is happening use Sequential, but if HNSW graph is building/searching then flip back to RANDOM). I am not sure what could be a consequence of this would like to know what community thinks about it. Similar to option 4. Also, I do believe that since Lucene99FlatVectorsFormat now extends KNNVectorsFormat thanks to this PR: https://github.com/apache/lucene/pull/13469, having an ability to change the madvise from consumers of this format is needed. So that Lucene99FlatVectorsFormat can be used independently and not always tied with HNSW. ### FAQ 1. On what machines the benchmarks were performed with 10M 768D dataset? Since Opensearch is distributed Lucene, the setup was like this: 1. 3 Opensearch nodes was used. 2. All nodes had 32GB of Heap 3. All nodes had 128GB of RAM and 16 vCPUs. 4. The 10M dataset was divided in 6 lucene indices aka primary shards. So per lucene index there was 1.6M docs. 5. Only vector field was indexed. 7. Is the benchmarks performed on Lucene library independently? No, we have not performed any benchmarks with Lucene library independently, but I am working on having a reproducible setup. If there are some easy way to setup and reproduce, please share. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
