mikemccand opened a new issue, #15901:
URL: https://github.com/apache/lucene/issues/15901

   ### Description
   
   I'm looking at a fun segment trace of a nearly pure flat (no HNSW graph) 
vectors Lucene index (production index for Amazon product search) and see 
weirdly that the flushed segments show ~112% RAM efficiency.
   
   RAM efficiency measures size-on-disk of the newly flushed segment against 
size-in-ram before it was flushed.  For postings heavy indices it's usually 
poor, maybe 20-30%, because in-RAM postings storage has overhead because they 
need to grow/realloc as new documents are indexed, appending integers to tons 
of tiny `int[]` virtual slices (one or two or three per unique term in the 
document being inverted).  On disk those become well packed.
   
   But for flat vectors, I think we just buffer the vectors in RAM until flush, 
so the RAM accounting ought to be simple?
   
   This is with 7-bit quantization (aside: why do we now have signed 7 bit, and 
unsigned 8 bit, `KnnVectorsFormat` options?  is there really a difference?  oh 
hmm the javadocs says its for backwards compatibility ... shouldn't it (7 bit) 
be marked deprecated (new Lucene indices shouldn't choose it)?), and @msokolov 
pointed out we increase index size to store the one-byte-per-dimension 
quantized vectors in addition to the `float32` vectors.  But then I would have 
expected 125% RAM efficiency?  So I'm confused why I see halfway (~112%)...
   
   <img width="730" height="530" alt="Image" 
src="https://github.com/user-attachments/assets/589ac3c8-b390-415f-8c8d-2f6ec68d752c";
 />
   
   ### Version and environment details
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to