gaoj0017 commented on PR #14078: URL: https://github.com/apache/lucene/pull/14078#issuecomment-2664639299
As we have consistently emphasized in both public and private communications, we are concerned that the <ins>**OSQ method employs an idea highly similar to the one presented in our [extended RaBitQ paper](https://arxiv.org/pdf/2409.09913), which was published months earlier than OSQ - however, Elastic does not acknowledge extended RaBitQ as a prior art but as a parallel work**</ins>. Specifically, the high-level idea of OSQ is to try different parameters for scalar quantization on a per-vector basis as described in Elastic’s reply. This idea is highly similar to one of the key ideas of our extended RaBitQ (as described in Section 3.2.2 of our extended RaBitQ paper): it involves testing various parameters for each individual vector, performing rounding (i.e., scalar quantization) based on these parameters, and then calculating the similarity between the rounded vector and the original vector. It finally selects the optimal parameter and corresponding rounded vector. With this, ou r extended RaBitQ method achieves significant performance improvements over LVQ, the state-of-the-art variant of scalar quantization by then. For example, under 2-bit quantization, the distance estimation error of LVQ is larger than that of the extended RaBitQ by orders of magnitude. In addition, Elastic insists on categorizing our RaBitQ and its extension methods as a variant of PQ to highlight the differences from OSQ, despite our repeated refutation of this point in our responses in private meetings. We believe we have made it clear that our RaBitQ and its extension are not variants of PQ, but provide optimized methods of binary and scalar quantization, respectively. We respect OSQ’s differences from extended RaBitQ and would be pleased to see if they can make further improvements. However, these differences do not negate the similarity between the two methods. At the same time, we hope Elastic will respect (1) the significant performance improvements brought by our extended RaBitQ method in the realm of scalar quantization and (2) the fact that our extended RaBitQ method was published months earlier than OSQ. Finally, we would like to reiterate our two points: 1. <ins>**The so-called BBQ majorly follows our RaBitQ method;**</ins> 2. <ins>**The OSQ method (introduced in this PR) has its major idea similar to our extended RaBitQ method (published months earlier) and our extended RaBitQ method is a prior art which achieves good accuracy at 1-bit/2-bit binary/scalar quantization for the first time.**</ins> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
