gautamworah96 commented on PR #13553:
URL: https://github.com/apache/lucene/pull/13553#issuecomment-2218434337
Interesting. I didn't know the ram usage estimator is what is actually used
for flushing based on ram size. I used to think it was only for external
monitoring/alarming, and some o
benwtrent merged PR #13553:
URL: https://github.com/apache/lucene/pull/13553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.a
benwtrent commented on PR #13553:
URL: https://github.com/apache/lucene/pull/13553#issuecomment-2215645063
@gautamworah96 @msokolov this might be part of the reason for the OOMs, the
estimates were completely ignoring the float[] vector sizes for fieldwriters 🤦
. I plan on iterating on this
benwtrent commented on code in PR #13553:
URL: https://github.com/apache/lucene/pull/13553#discussion_r1669465330
##
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsWriter.java:
##
@@ -299,9 +299,7 @@ public void finish() throws IOException {
benwtrent commented on code in PR #13553:
URL: https://github.com/apache/lucene/pull/13553#discussion_r1669464616
##
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsWriter.java:
##
@@ -172,9 +172,6 @@ public void finish() throws IOException {
public
benwtrent opened a new pull request, #13553:
URL: https://github.com/apache/lucene/pull/13553
I still need to write a test, but wanted to open this PR early.
Scalar Quantized vector writer ram usage estimates completely ignores the
raw float vectors. Meaning, if you have flush based o
