msokolov commented on PR #14874: URL: https://github.com/apache/lucene/pull/14874#issuecomment-3057200869
I did some deep-diving with profiler and I realized that when indexing, we call these dotProduct methods in a different context in which all of the vectors are on-heap. I'm surmising that this induces hotspot compiler to make some choices that end up being bad later when half of the vectors (the document ones) are off-heap; ie while searching. To test this theory I cloned the `dotProduct(ByteSegment, ByteSegment)` and called the new `dotProductWTF(ByteSegment, ByteSegment)` from `dotProduct(byte[], byte[])`. This did indeed lead to even greater speedups (another 2x, astoundingly), although strangely, a small but significant drop in recall?? I'm now trying again with single-threaded merges to see if somehow maybe increased lock contention during merging is leading to the recall drop. Hmm actually I wonder if it's simply that there are a different number of segments being produced. Separately, this whole line of attack made me wonder if maybe we ought to be storing our vectors differently while indexing ... instead of a `List<byte[]>` or `List<float[]>` we might prefer to preallocate a larger array and write to it, advancing a pointer as we go. If we did this we could also maybe preallocate a MemorySegment along with it to use for scoring??. I don't know what impact if any this might have on the whole hotspot discovery, but it is becoming clear to me that the way we manage memory can have a major impact on the performance here in nonobvious ways. I'm also still somewhat confused about the status of the whole Paname vectorization code. I guess it is still incubating, so we have decided not to require it, and our code paths are conditional on whether it is enabled or not. I think this conditionalization itself may be problematic from a performance perspective, although this is more a suspicion than anything, but I wonder if it would be possible to just require users to enable the incubating API be enabled at this point? It is in its umpteenth uncubation cycle, seems like it should be safe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
