iverase commented on code in PR #16571:
URL: https://github.com/apache/lucene/pull/16571#discussion_r4399863870
##########
lucene/core/src/java/org/apache/lucene/index/MergePolicy.java:
##########
@@ -327,6 +357,73 @@ public Sorter.DocMap reorder(CodecReader reader, Directory
dir, Executor executo
return null;
}
+ /**
+ * Expert: split this merge into several output segments, each holding a
contiguous doc range of
+ * every input.
+ *
+ * <p>Returns {@code null} (the default) for the usual behaviour of
producing exactly one
+ * segment. Called once, <b>after</b> merge readers have been opened, so
an implementation may
+ * inspect {@link #getMergeReader()} to place boundaries on real key
values rather than on doc
+ * counts. Otherwise the returned array has one entry per input segment,
in the order of {@link
+ * #segments}. Each entry is a non-decreasing array of {@code outputCount
+ 1} boundaries
+ * starting at {@code 0} and ending at that input's {@code maxDoc}: output
{@code o} takes docs
+ * {@code [b[o], b[o+1])} from that input.
+ *
+ * <p><b>Cost:</b> bytes written are unaffected -- every document is
written exactly once, into
+ * whichever output owns it. Reads depend on whether a format can skip by
document range. The
+ * inputs are verified once for the whole merge rather than once per
output, and doc values seek
+ * to the range an output owns, so those cost about what an ordinary merge
costs.
+ *
+ * <p>Two formats cannot skip, and each output reads them in full: a terms
dictionary is ordered
+ * by term, so a term's postings are spread across every output and
masking documents saves
+ * nothing; and a block k-d tree is ordered by value, so no leaf can be
skipped by document
+ * range. Both are linear in the output count. Stored fields and term
vectors are read by random
+ * access per document, so an output touches only the chunks its own range
falls in, plus the
+ * two it straddles -- a constant penalty rather than one that grows with
the output count.
+ *
+ * <p>Note that splitting a merge does not reduce the segment count the
way an ordinary merge
+ * does, so a policy that partitions also has to choose merges whose
inputs belong to the same
+ * part of the key space; otherwise the same data is merged apart and back
together repeatedly.
+ *
+ * <p>Expressing the split as boundaries rather than arbitrary doc sets
makes the two properties
+ * {@link IndexWriter} depends on structural rather than a caller
obligation: the outputs are
+ * <b>disjoint</b> and together <b>cover every document</b>. {@link
IndexWriter} validates the
+ * shape and rejects the merge otherwise.
+ *
+ * <p><b>Wrapping policies must delegate this method and {@link
#isPartitioned()}.</b> A wrapper
+ * such as {@link OneMergeWrappingMergePolicy} that re-wraps the {@link
OneMerge} without
+ * forwarding them silently downgrades the merge to a single output -- no
error, just no
+ * partitioning.
+ *
+ * <p>An index sort is not required, but it is what makes an output mean
something: because
+ * every input is then sorted by the same key, a contiguous doc range is a
contiguous key range,
+ * so each output holds a range of keys and is itself correctly sorted.
Without one the split is
+ * still well defined -- the documents are simply distributed by position.
+ *
+ * @lucene.experimental
+ */
+ public int[][] getDocRangePartitions(List<CodecReader> readers) throws
IOException {
Review Comment:
It took me a while to understand the contents of the return type of this
method. Every entry of the array is the partition for a segment. So the size of
the array must be size of the readers and the length of the partition arrays
should be the same.
I wonder if we could add a class that makes this more explicit or is too
much overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
