LantaoJin opened a new pull request, #16879: URL: https://github.com/apache/lucene/pull/16879
### Description <!-- If this is your first contribution to Lucene, please make sure you have reviewed the contribution guide. https://github.com/apache/lucene/blob/main/CONTRIBUTING.md --> When a merge policy reorders a merge (for example `BPReorderingMergePolicy`), `IndexWriter` hands the codec a single `SortingCodecReader` over a `SlowCompositeCodecReaderWrapper`. Neither wrapper's vectors reader is a `HnswGraphProvider`, so `IncrementalHnswGraphMerger` finds no graph to reuse and rebuilds the merged graph from scratch. The same happens for `addIndexes(SortingCodecReader.wrap(...))`, which is how `BpVectorReorderer` reorders an existing index (#13565 noted that this "recreates the HNSW graph doing unnecessary work, when all we really want to do is to renumber it"). The source graphs are still valid: `SortingCodecReader`'s vector values keep the ordinals of the wrapped reader, and the composite reader's ordinals are the concatenation of its sub-readers'. The merger only lacks a way to reach the wrapped readers. This PR adds a small experimental interface, `HnswGraphMergeSources`, that a view implements to return the readers it wraps together with their doc ID mapping. Both wrappers implement it, and `IncrementalHnswGraphMerger.addReader` adds those readers in place of the view, composing the doc maps and mapping the view's live docs back to each reader. `ConcurrentHnswMerger` inherits this. Merge time for 80k 64-dim vectors, force-merged with a merge policy that reverses doc order (single merge worker, one run each): | Shape | Before | After | Same merge, not reordered | |---|---|---|---| | 8 segments x 10k | 23.7 s | 16.9 s | 16.7 s | | 60k + 4 x 5k | 23.1 s | 6.3 s | 7.5 s | | 1 segment x 80k, 5% deleted | 22.5 s | 2.4 s | 2.2 s | Recall@10 (fanout 100) after the fix matches the not-reordered merge in all three cases. `TestHnswGraphReuseDuringReorderedMerge` covers a reordered multi-segment merge, a reordered singleton merge with deletions (serial and concurrent mergers), `addIndexes` of a sorted deletion-free segment (checking that the merged graph is exactly the source graph with renumbered nodes), and `addIndexes` of a filtered sorted view over two sparse segments. All five tests fail on main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
