iprithv opened a new pull request, #15982: URL: https://github.com/apache/lucene/pull/15982
### Description Vector RAM accounting in `ramBytesUsed()` had three bugs causing IndexWriter to undercount memory usage for buffered vectors, leading to delayed flush decisions and higher than expected memory consumption. ### Bugs Fixed Fixes #15901 **1. `BufferingKnnVectorsWriter` hardcoded `Float.BYTES` for all encodings** Byte vectors (`VectorEncoding.BYTE`) were reported as 4x their actual size because `ramBytesUsed()` always multiplied by `Float.BYTES` (4) instead of `Byte.BYTES` (1). This is technically an *overcount* for byte vectors, but it's wrong in the opposite direction, it masks the undercounting elsewhere and produces incorrect flush thresholds. **2. Quantized writers never counted `rawVectorDelegate` RAM** `Lucene104ScalarQuantizedVectorsWriter`, `Lucene99ScalarQuantizedVectorsWriter`, and `Lucene102BinaryQuantizedVectorsWriter` all wrap a `rawVectorDelegate` (`Lucene99FlatVectorsWriter`). For FLOAT32 fields, the delegate's field-level data was counted indirectly through `FieldWriter.flatFieldVectorsWriter.ramBytesUsed()`. But for BYTE fields, which bypass the quantized `FieldWriter` entirely, the delegate was never queried, making byte vector RAM completely invisible (48 bytes reported for hundreds of KB of actual data). Refactored all three writers to call `rawVectorDelegate.ramBytesUsed()` at the writer level for all flat vector data, and `quantizationOverheadBytesUsed()` for quantization-specific state (magnitudes, dimensionSums) to avoid double-counting. **3. `dimensionSums` array not counted** The `float[dimension]` array used for centroid calculation during flush was not included in `ramBytesUsed()` for `Lucene104ScalarQuantizedVectorsWriter` and `Lucene102BinaryQuantizedVectorsWriter`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
