Pulkitg64 commented on code in PR #15003:
URL: https://github.com/apache/lucene/pull/15003#discussion_r2254248192
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsWriter.java:
##########
@@ -347,12 +350,195 @@ private void reconstructAndWriteNeighbours(
}
}
+ /**
+ * Processes an off-heap HNSW graph, removing deleted nodes and writing the
graph structure to the
+ * vector index.
+ *
+ * @param graph The off-heap graph to process
+ * @param docMap Mapping from old document IDs to new document IDs
+ * @param graphSize The size of the graph (number of vectors)
+ * @param offsets Array to store the byte offsets for each node at each level
+ * @return A mock HnswGraph implementation that provides access to the graph
structure
+ * @throws IOException If an error occurs while writing to the vector index
+ */
+ private HnswGraph deleteNodesWriteGraph(
+ Lucene99HnswVectorsReader.OffHeapHnswGraph graph,
+ MergeState.DocMap docMap,
+ int graphSize,
+ int[][] offsets)
+ throws IOException {
+ if (graph == null) return null;
+
+ int[] scratch = new int[graph.maxConn() * 2];
+ final int numLevels = graph.numLevels();
+ final int[][] validNodesPerLevel = new int[numLevels][];
+
+ // Process all levels
+ for (int level = 0; level < numLevels; level++) {
+ int[] sortedNodes =
NodesIterator.getSortedNodes(graph.getNodesOnLevel(level));
+
+ // Count and collect valid nodes
+ int validNodeCount = 0;
+ for (int node : sortedNodes) {
+ if (docMap.get(node) != -1) {
+ validNodeCount++;
+ }
+ }
+
+ // Special case for top level with no valid nodes
+ if (level == numLevels - 1 && validNodeCount == 0 && level > 0) {
+ validNodeCount = 1; // We'll create one connection to lower level
+ }
+
+ validNodesPerLevel[level] = new int[validNodeCount];
+ offsets[level] = new int[validNodeCount];
+
+ int validNodeIndex = 0;
+ int nodeOffsetId = 0;
+
+ // Process nodes at this level
+ for (int node : sortedNodes) {
+ if (docMap.get(node) == -1) {
Review Comment:
This is incorrect. Graph does not store docIDs but instead they store
ordinal. Whereas docMap maps oldDocIds to new DocIDs.
The correct implementation is to create a map which maps old ords to new
ords.
Will fix this in next revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
