jeho-rpls opened a new issue, #16367:
URL: https://github.com/apache/lucene/issues/16367
### Description
### Summary
`IndexWriter#abortMerges()` sets the abort flag on running merges and waits
for them to finish, relying on merges observing the flag periodically (the
comment in `abortMerges` says *"It should not take very long because they
periodically check if they are aborted."*). That assumption holds for I/O-bound
merge phases — every write goes through the merge rate limiter, which checks
for abort — and since #16264 / #16281, `checkIntegrity()` is also covered.
HNSW graph construction breaks this assumption:
`HnswGraphBuilder#addVectors` is a pure-CPU loop that performs no writes and
never checks `OneMerge#isAborted()`. For large vector segments this phase runs
for **10–25 minutes single-threaded**, during which `rollback()` /
`abortMerges()` block unconditionally.
We believe this is now the last significant blind spot in merge abort
handling, and it is the dominant one for vector-heavy indexes: in our
measurements graph construction is **93.6% of merge CPU**, while the checksum
phase addressed by #16281 is 0.1%.
### Production impact
Elasticsearch 9.4.2 (Lucene 10.4.0) cluster, HNSW-indexed dense vectors (1M
× 384d float32 + 575k × 96d quantized per shard). During a routine rolling
restart, each data node's shutdown blocked on `abortMerges` waiting for an
in-flight HNSW merge, for **24–31 minutes per node** — exceeding the 30-minute
termination grace period, so pods had to be force-killed.
Aborting thread:
```
"elasticsearch[...][generic][T#…]" TIMED_WAITING
at java.lang.Object.wait0(java.base/Native Method)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:5571)
at org.apache.lucene.index.IndexWriter.abortMerges(IndexWriter.java:2760)
at
org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2514)
at
org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2483)
```
The single merge thread it waits for (RUNNABLE the whole time, no writes):
```
"elasticsearch[...][merge][T#3]" RUNNABLE
... (vector scoring frames elided)
at
org.apache.lucene.util.hnsw.HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:342)
at
org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNodeInternal(HnswGraphBuilder.java:312)
at
org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNode(HnswGraphBuilder.java:354)
at
org.apache.lucene.util.hnsw.HnswGraphBuilder.addVectors(HnswGraphBuilder.java:233)
at
org.apache.lucene.util.hnsw.HnswGraphBuilder.build(HnswGraphBuilder.java:201)
at
org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.merge(IncrementalHnswGraphMerger.java:218)
at
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.mergeOneField(Lucene99HnswVectorsWriter.java:449)
at
org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:126)
at
org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:105)
at
org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:271)
...
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5323)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4784)
```
### Measurements
Instrumenting one incident-class merge (TieredMergePolicy deletes-purging
rewrite of a ~3 GB segment, 1.9M docs, JFR + async-profiler):
| Phase | Wall time | Abort-checked today? |
|---|---|---|
| checkIntegrity (checksums) | seconds (0.1% CPU) | yes on main (#16264 /
#16281) |
| stored fields / postings / doc values / vector data copy | ~8 min | yes
(rate limiter on writes) |
| **HNSW graph construction (both fields)** | **~14.5 min** | **no** |
| HNSW graph serialization + segment finish | ~1 min | yes (writes) |
| total | 23.5 min | |
Graph construction = 93.6% of merge CPU samples (higher than its wall-time
share because the copy phases are I/O-bound). `main` has the same shape:
`HnswGraphBuilder#addVectors` contains no abort/cancellation hook.
A standalone reproduction below (lucene-core only, 160k × 384d vectors):
`rollback()` called 8s into a `forceMerge` blocks for **68.6s** on stock 10.4.0
— the entire remaining graph build.
<details>
<summary>Reproduction (lucene-core 10.4.0 on the classpath, ~1 min indexing
+ ~1 min to observe the block)</summary>
```java
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Random;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.KnnFloatVectorField;
import org.apache.lucene.index.ConcurrentMergeScheduler;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.NoMergePolicy;
import org.apache.lucene.store.FSDirectory;
/** rollback() while an HNSW merge is building its graph blocks until the
build finishes. */
public class MiniRepro {
public static void main(String[] args) throws Exception {
int dim = 384, segs = 16, perSeg = 10000;
long sleepMs = args.length > 0 ? Long.parseLong(args[0]) : 8000;
Path dir = Files.createTempDirectory("hnsw-abort-repro");
try (FSDirectory d = FSDirectory.open(dir)) {
IndexWriterConfig cfg = new IndexWriterConfig();
cfg.setUseCompoundFile(false);
cfg.setMergePolicy(NoMergePolicy.INSTANCE);
try (IndexWriter w = new IndexWriter(d, cfg)) {
Random r = new Random(42);
for (int s = 0; s < segs; s++) {
for (int i = 0; i < perSeg; i++) {
Document doc = new Document();
float[] v = new float[dim];
for (int j = 0; j < dim; j++) v[j] = r.nextFloat();
doc.add(new KnnFloatVectorField("v", v));
w.addDocument(doc);
}
w.flush();
}
w.commit();
}
IndexWriterConfig cfg2 = new IndexWriterConfig();
cfg2.setMergeScheduler(new ConcurrentMergeScheduler());
IndexWriter w2 = new IndexWriter(d, cfg2);
long m0 = System.nanoTime();
Thread bg = new Thread(() -> {
try {
w2.forceMerge(1);
System.out.printf("[bg] forceMerge completed normally in %.1fs%n",
(System.nanoTime() - m0) / 1e9);
} catch (Throwable t) {
System.out.printf("[bg] forceMerge exception after %.1fs: %s%n",
(System.nanoTime() - m0) / 1e9, t);
}
});
bg.start();
Thread.sleep(sleepMs);
System.out.printf("calling rollback() at t=%.1fs after merge start%n",
(System.nanoTime() - m0) / 1e9);
long r0 = System.nanoTime();
w2.rollback();
System.out.printf("### rollback() returned in %.2fs ###%n",
(System.nanoTime() - r0) / 1e9);
bg.join();
}
}
}
```
Observed on 10.4.0 (Apple M-series, JDK 23, `--add-modules
jdk.incubator.vector`):
```
calling rollback() at t=8.0s after merge start
### rollback() returned in 68.63s ###
```
With a periodic abort check added to `HnswGraphBuilder#addVectors` (same
machine, same seed):
```
calling rollback() at t=8.0s after merge start
[bg] forceMerge exception after 8.2s: ... MergeAbortedException ... [ABORTED]
### rollback() returned in 0.20s ###
```
Note: on 10.4.0 this reproduction always takes the full-rebuild path
(`HnswGraphBuilder#addVectors`), because
`PerFieldKnnVectorsFormat.FieldsReader` does not implement `HnswGraphProvider`
there, so graph reuse never kicks in under the default codec. On `main`, where
the wrapper does implement it, clean segments take the
`MergingHnswGraphBuilder` join path instead — which also has no abort checks
(see step 3 below), so the repro demonstrates the block either way.
</details>
### Prior reports
- #11694 (LUCENE-10658, 2022): general request for abortable merges; still
open. Elasticsearch's side (elastic/elasticsearch#88055) was closed pointing to
it.
- #13354: same pathology, closed after #16264 / #16281 made *checksumming*
abortable — which is 0.1% of this workload.
- #13822: proposed plumbing `OneMergeProgress` into `MergeState` for exactly
this scenario (HNSW stack in the report), but received no review and its
plumbing is now superseded by `MergeState#oneMerge` (#16264).
### Proposed fix
Extend the pattern established by #16264 / #16281 to the graph build path:
1. The plumbing already exists on main: `MergeState#oneMerge` and the
null-safe `MergeState#checkAborted()` (#16264), which `KnnVectorsWriter#merge`
already invokes once per reader before merging. What's missing is handing that
same check down from `Lucene99HnswVectorsWriter#mergeOneField` through the HNSW
graph merger into the builders.
2. In `HnswGraphBuilder#addVectors`, invoke it periodically — e.g. every
1024 nodes; each node insertion costs hundreds of vector comparisons, so the
overhead is unmeasurable. (This and step 3 could likely be consolidated into a
single check in the shared node-insertion path,
`addGraphNode`/`addGraphNodeInternal` — happy to follow whichever reviewers
prefer.)
3. Cover the incremental-merge path too (`MergingHnswGraphBuilder#build`'s
remainder-insertion loop and `updateGraph`), which inserts via `addGraphNode`
directly.
4. Same null contract as `checkIntegrity(OneMerge)`: flush-time graph builds
pass null and are unaffected.
We validated the approach with a patched 10.4.0 in the affected cluster
(same periodic check in `addVectors`; wired through a ThreadLocal shim only
because 10.4 predates `MergeState#oneMerge` — the proposal above uses the
explicit plumbing instead): node shutdown during a live graph build went from
24–31 min (grace exceeded, force kill) to **2.2 s** (measured between the
node's "stopping" and "stopped" log lines), aborted merges clean up through the
normal `MergeAbortedException` path, and undisturbed merges complete unchanged.
Happy to contribute a PR with a test (rollback during HNSW merge must abort
promptly) if this direction sounds right.
### Version and environment details
```
Lucene 10.4.0 (bundled in Elasticsearch 9.4.2); addVectors on main also
lacks abort checks.
OpenJDK 26.0.1, Linux aarch64, single-threaded merges (no intra-merge
executor).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
