msfroh commented on a change in pull request #1155: LUCENE-8962: Add ability to
selectively merge on commit
URL: https://github.com/apache/lucene-solr/pull/1155#discussion_r368260601
##########
File path: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
##########
@@ -3147,6 +3149,38 @@ public final boolean flushNextBuffer() throws
IOException {
}
}
+ private MergePolicy waitForMergeOnCommitPolicy(MergePolicy source, final
SegmentInfos toCommit,
+
AtomicReference<CountDownLatch> mergeLatchRef) {
+ return new OneMergeWrappingMergePolicy(source, (toWrap) -> new
MergePolicy.OneMerge(toWrap.segments) {
+ @Override
+ public void mergeFinished() throws IOException {
+ super.mergeFinished();
+ CountDownLatch mergeAwaitLatch = mergeLatchRef.get();
+ if (mergeAwaitLatch == null) {
+ // Commit thread timed out waiting for this merge and moved on. No
need to manipulate toCommit.
+ return;
+ }
+ if (isAborted() == false) {
+ deleter.incRef(this.info.files());
+ toCommit.add(this.info.clone());
+ long segmentCounter =
Long.parseLong(this.info.info.name.substring(1), Character.MAX_RADIX);
+ toCommit.counter = Math.max(toCommit.counter, segmentCounter + 1);
+ Set<String> segmentNamesToRemove = new HashSet<>();
+ for (SegmentCommitInfo sci : this.segments) {
+ deleter.decRef(sci.files());
+ segmentNamesToRemove.add(sci.info.name);
+ }
+ for (int i = toCommit.size() - 1; i >= 0; i--) {
+ if (segmentNamesToRemove.contains(toCommit.info(i).info.name)) {
+ toCommit.remove(i);
Review comment:
I've now forgotten my motivation for traversing the list backwards. I think
it was just to avoid needing to do some potentially messy/error-prone index
manipulation.
In my latest commit, I realized that I could leverage
`SegmentInfos.applyMergeChanges` if I replace the merged segments with their
`toCommit` doppelgängers. I think that provides better encapsulation and lets
`SegmentInfos` manage its own internal state better. (Also `applyMergeChanges`
does some weird and wonderful list manipulation of its own that I definitely
didn't want to replicate into `IndexWriter`.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
