mikemccand commented on PR #15903: URL: https://github.com/apache/lucene/pull/15903#issuecomment-4169876424
Wow, what an impressive genai example! I also know nearly nothing about TQ, and only scratch surfaces in understanding OSQ. I am curious how the two compare. E.g. does OSQ also not alter the quantization per-segment (merge of flat vectors could optimized `copyBytes` (the hardest function in the world to implement correctly/performantly!))? Do we get a 3 bit option with OSQ? Thank you @xande for preserving the iterations (separate commits) as you stepped through the plan with Kiro. Is the original plan/prompting visible somewhere here? I wish we all would preserve all prompts/plans/soul context docs -- they should be treated like source code. Imaging finding an exotic bug in this Codec some time in the future and being able to look back at how the prompts were written, how Kiro iterated, etc., to gain insight. Also, it would help us all learn how to use genai if we were better about sharing prompts / steering docs. Today, genai is a lonely endeavor -- what little human contact we had in a team / our craft is being replaced with solo time with your genai. Kinda like putting on your Apple Vision Pro. Genai is missing good tooling/culture to enable human to human collaboration/learning. I'd love to see ROC-type curves using luceneutil's `knnPerfTest.py`, showing tradeoff of latency vs recall as you turn the "try harder" query time knob (oversample? fanout?) for all of Lucene's vector codecs (core OSQ/BBQ/etc., but also including Faiss and JVector!). [A side rant: it's weird that nobody talks about precision of our vector queries, I guess because that's a lot more work to measure (you need an annotated corpus that marks pairs of query/index vectors with at least relevant/irrelevant binary classification), and, it's really measuring the model that generated the embeddings. So, we drastically simplify, assume the model is perfection, all vectors are precisely relevant if they are close, and only measure recall.] > Thats the last run @mccullocht did for Lucene's OSQ technique (1M vectors, would need to do the exact same data set for Apples to apples). +1 to do as apples/apples comparison as we can. But what corpus was this @benwtrent? (@mccullocht later mentioned [voyage-3.5](https://blog.voyageai.com/2025/05/20/voyage-3-5/) but I want to confirm the results you listed). It's interesting how different each corpus is -- I wish there were some way to visualize these massive-dimension vectors. High dimension math can be [crazy counterintuitive](https://www.youtube.com/watch?v=fsLh-NYhOoU)! > I realize performance apples to apples will take way more work (panama vector APIs, etc.). I am more concerned about recall, and I am not sure TQ will provide any significant recall improvement itself. I think recall, total CPU, wall-clock-time-with-many-cores, and effective hot RAM required (e.g. 2nd reranking phase is a big penalty there) at query time, and then also indexing performance, are all important when comparing the many vector Codecs we have now. Maybe we can just submit a bunch of competitors to [ann-benchmarks](https://ann-benchmarks.com/index.html)? Hmm maybe one can run their own ann-benchmarks instance (using their [GitHub repo](https://github.com/erikbern/ann-benchmarks/))? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
