tlrx opened a new pull request, #16620:
URL: https://github.com/apache/lucene/pull/16620
We observed some merge threads pinned at 100% CPU building a HNSW graph
while the Lucene index writer rollback was blocked waiting for the merges to
abort:
```java
...
app/[email protected]/org.apache.lucene.util.hnsw.HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:342)
app/[email protected]/org.apache.lucene.util.hnsw.InitializedHnswGraphBuilder.fixDisconnectedNodes(InitializedHnswGraphBuilder.java:323)
app/[email protected]/org.apache.lucene.util.hnsw.InitializedHnswGraphBuilder.repairDisconnectedNodes(InitializedHnswGraphBuilder.java:279)
app/[email protected]/org.apache.lucene.util.hnsw.InitializedHnswGraphBuilder.initializeFromGraph(InitializedHnswGraphBuilder.java:192)
app/[email protected]/org.apache.lucene.util.hnsw.InitializedHnswGraphBuilder.fromGraph(InitializedHnswGraphBuilder.java:121)
app/[email protected]/org.apache.lucene.util.hnsw.InitializedHnswGraphBuilder.initGraph(InitializedHnswGraphBuilder.java:147)
app/[email protected]/org.apache.lucene.util.hnsw.MergingHnswGraphBuilder.fromGraphs(MergingHnswGraphBuilder.java:104)
app/[email protected]/org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.createBuilder(IncrementalHnswGraphMerger.java:175)
app/[email protected]/org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.merge(IncrementalHnswGraphMerger.java:234)
app/[email protected]/org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.buildAndWriteGraph(Lucene99HnswVectorsWriter.java:488)
app/[email protected]/org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.lambda$mergeOneField$0(Lucene99HnswVectorsWriter.java:442)
app/[email protected]/org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter$$Lambda/0x00000000093a6818.run(Unknown
Source)
app/[email protected]/org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:146)
app/[email protected]/org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:273)
app/[email protected]/org.apache.lucene.index.SegmentMerger$$Lambda/0x00000000093a55d0.merge(Unknown
Source)
app/[email protected]/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:316)
app/[email protected]/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:160)
app/[email protected]/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5400)
app/[email protected]/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4859)
app/[email protected]/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6684)
...
```
The abort check added in #16368 is only consulted during the incremental
node insertion path (ie, `addGraphNode` method) but not during the graph
initialization and repair phases (methods `copyGraphStructure`,
`repairDisconnectedNodes`/`fixDisconnectedNodes`, `rebalanceGraph`) which
happen before.
This change threads the abort check down to the graph initialization and
repair methods so that those phases can also be promptly aborted if a merge is
cancelled.
The `abortCheck` was introduced as non-final so I had to workaround this a
bit. I also kept the public static methods without the `abortCheck` parameter
around in the hope that it could help backporting this to 10.6 / 10.5.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
