mccullocht commented on PR #15903: URL: https://github.com/apache/lucene/pull/15903#issuecomment-4201716748
> TQ-1bit, as implemented in my poc branch, only stores quantized data which is 539 MB total (30x less). Today there's no way to opt out of float32 storage in Lucene's quantized formats (What are the reasons for that? I assume because it needs to keep float32 vectors around for requantization during segment merges?). OSQ centers the vectors -- it computes a mean vector within the segment, then quantizes the residual vector `v - c`. When building a new segment this operation is repeated. Lucene104SQFlatVectorsFormat could be pretty easily extended to use a zero vector instead of computing a mean and make merging a byte copy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
