kaivalnp commented on PR #15979: URL: https://github.com/apache/lucene/pull/15979#issuecomment-4315304229
> looks as if the deduplicated vectors PR is actually a bit faster?! Yeah, this was surprising to me too -- it is \~10% faster, though I'm not sure why (there's one additional lookup of `vector ord -> ord of position in data file` which is stored on-heap as an `int[]`). > How do you think this would combine with all the other vector formats, such as the quantizing ones? This is kind of tricky -- the main challenge is that quantization factors can depend on data distribution? (i.e. the same `float[]` vector may be quantized to a different `byte[]` for smaller graphs). A slightly smaller challenge is that quantization formats today store the quantized `byte[]` vector followed by a `float` correction factor on disk -- and because both are read at the same time, this layout leads to fewer page cache misses. For de-duplication to be effective, I guess we'd need to decouple the quantized `byte[]` vector from the `float` correction factor? This would lead to more page cache misses -- but FWIW the correction factors are just 4B per vector per field, and can be "hot" too (either explicitly: by keeping on-heap, or implicitly: like HNSW graph edges, they are an order of magnitude smaller). > do you see this as becoming the default flat vector format IMO it could be the default if the overhead of de-duplication is "small" (personally: something like <10% slower indexing / merge sounds like it would be worth replacing the default). Since it is currently not: perhaps a separate HNSW format backed by the de-duplicating format in a non-core module? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
