shatejas commented on PR #13985: URL: https://github.com/apache/lucene/pull/13985#issuecomment-3201592909
> Perhaps if we have a case where there is no random access (from Lucene) at all and we are only using Lucene to store the vector data - any searh indexing is being done by a native plugin (I think this is what you are targeting?) then we don't really want to be switching back and forth between access modes In certain scenarios, random access is critical for optimal performance. Our initial hypothesis, which we are currently validating through benchmarks, is that random access lookups are significantly more efficient than sequential lookups under high memory pressure, especially when doing a exact search on filtered documents. This is because random access avoids data prefetching, thereby reducing memory swapping. To provide users with greater control, its best to allow them to configure initial IOContext value based on their specific workloads, rather than keeping it a fixed constant. This approach will offer flexibility while maintaining sensible defaults. Furthermore, we can mitigate the impact on search performance during merges by implementing a dedicated prefetch functionality for vector merges. This eliminates the need to switch between access methods, ensuring a minimal impact on ongoing searches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
