benwtrent commented on PR #14078: URL: https://github.com/apache/lucene/pull/14078#issuecomment-2679008833
@gaoj0017 > The OSQ method (introduced in this PR) has its major idea similar to our extended RaBitQ method and our extended RaBitQ method is a prior art which achieves good accuracy at 1-bit/2-bit binary/scalar quantization for the first time. Thank you for further clarifying what y’all are meaning by prior art, this helps us understand each of our “sticking” points. Extended RaBitQ is, to our knowledge, the first thing to do something other than min/max quantization per vector. Indeed, extended RaBitQ has impressive numbers from y’all’s paper and work. Our sticking point is that the quantization optimizations for OSQ are unique. While both are improving the quantization quality per vector, OSQ is going about it in a different way. Its direct scalar quantile optimization via anisotropic loss. Both of which find their roots in the linked research in the PR. I will happily indicate in the code comments that extended RaBitQ is the first to do this more advanced quantization per vector. Because to our knowledge it is true. We want to donate this to the Apache Lucene project and community so it can grow and evolve here. Additionally, we are wanting to do a broader survey work (though likely not an academic article), that shows the strengths and improvements over all of the major quantization techniques, including extended RaBitQ, LVQ, PQ, and OSQ. I think a direct comparison between the techniques helps provide color for the strengths of each. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
