shbhar commented on PR #15903: URL: https://github.com/apache/lucene/pull/15903#issuecomment-4201569034
@mccullocht let me see if I can try this >I suggest you try rotating your vectors first and test recall with OSQ. It should be easy enough to perform the rotation outside of Lucene, and if there's significant value we can figure out if or how we'd like to internalize this. @tveasey I think the TQ-1bit vs BBQ-1bit comparison is misleading because the storage is very different. BBQ "1-bit" keeps the full float32 vectors alongside the binary quantization (16,178 MB at 4096d) and uses per-vector scalar correction terms during search. TQ-1bit, as implemented in my poc branch, only stores quantized data which is 539 MB total (30x less). Today there's no way to opt out of float32 storage in Lucene's quantized formats (What are the reasons for that? I assume because it needs to keep float32 vectors around for requantization during segment merges?). This TQ approach gives users the choice: if they want float32 reranking they can store vectors in a separate field and use a rescore query completely ignoring the dequant rescoring path - and the choice is meaningful because the built in rescore path does appear to have usable recall depending on dataset/dimensions. The recall gap also depends heavily on the dataset. Cohere and ASIN datasets have very different distributions (mean pairwise cosine similarity 0.23 vs 0.50), so comparing TQ-1bit recall across them isn't very informative. When we compare on the same dataset (100K MS MARCO passages, Qwen3-8B), TQ-1bit matches BBQ-1bit at 1024d (0.720 vs 0.721) and beats it at 4096d (0.807 vs 0.722). All at truly 1-bit storage with no float32 assistance. See my first post for the blessing of dimensionality section which has a test on multi-dimension on same dataset utilizing MRL - though I'm not sure if MRL property itself biases this comparison against BBQ somehow (MRL is increasingly common though) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
