mayya-sharipova commented on code in PR #14426:
URL: https://github.com/apache/lucene/pull/14426#discussion_r2027061812
##########
lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsReader.java:
##########
@@ -130,4 +134,56 @@ public KnnVectorsReader getMergeInstance() {
* <p>The default implementation is empty
*/
public void finishMerge() throws IOException {}
+
+ /** A string representing the off-heap category for quantized vectors. */
+ public static final String QUANTIZED = "QUANTIZED";
+
+ /** A string representing the off-heap category for the HNSW graph. */
+ public static final String HNSW_GRAPH = "HNSW_GRAPH";
+
+ /** A string representing the off-heap category for raw vectors. */
+ public static final String RAW = "RAW";
+
+ /**
+ * Returns the desired size of off-heap memory the given field. This size
can be used to help
+ * determine the memory requirements for optimal search performance, which
can be greatly affected
+ * by page faults when not enough memory is available.
+ *
+ * <p>For reporting purposes, the backing off-heap index structures are
broken into three
+ * categories: 1. {@link #RAW}, 2. {@link #HNSW_GRAPH}, and 3. {@link
#QUANTIZED}. The returned
+ * map will have zero or one entry for each of these categories.
+ *
+ * <p>The long value is the size in bytes of the off-heap space needed if
the associated index
+ * structure were to be fully loaded in memory. While somewhat analogous to
{@link
+ * Accountable#ramBytesUsed()} (which reports actual on-heap memory usage),
the metrics reported
+ * by this method are not actual usage but rather the amount of available
memory needed to fully
+ * load the index into memory, rather than an actual RAM usage requirement.
+ *
+ * <p>To determine the total desired off-heap memory size for the given
field:
+ *
+ * <pre>{@code
+ *
getOffHeapByteSize(field).values().stream().mapToLong(Long::longValue).sum();
+ * }</pre>
+ *
+ * @param fieldInfo the fieldInfo
+ * @return a map of the desired off-heap memory requirements by category
+ * @lucene.experimental
+ */
+ public abstract Map<String, Long> getOffHeapByteSize(FieldInfo fieldInfo);
Review Comment:
Very nice API! +1
But I am also thinking about other use cases: where we need to know the
total size across all vector fields and we don't know/don't remember field
names. Do you think it is worth to have this API: may be that returns a map of
maps?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
