javanna commented on code in PR #13542:
URL: https://github.com/apache/lucene/pull/13542#discussion_r1750502348
##########
lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java:
##########
@@ -363,7 +374,88 @@ public static LeafSlice[] slices(
LeafSlice[] slices = new LeafSlice[groupedLeaves.size()];
int upto = 0;
for (List<LeafReaderContext> currentLeaf : groupedLeaves) {
- slices[upto] = new LeafSlice(currentLeaf);
+ slices[upto] =
+ new LeafSlice(
+ new ArrayList<>(
+ currentLeaf.stream()
+ .map(LeafReaderContextPartition::createForEntireSegment)
+ .toList()));
+ ++upto;
+ }
+
+ return slices;
+ }
+
+ /**
+ * Creates leaf slices that leverage intra-segment concurrency by splitting
segments into multiple
+ * partitions according to the provided max number of documents per slice
and maximum number of
+ * segments per slice. If a segment holds more documents than the provided
max per slice, it gets
+ * split into equal size partitions that each gets its own slice assigned.
+ *
+ * <p>Note: this method is not yet called by the default slicing
implementation {@link
+ * #slices(List)}. Certain queries that require segment-level computation
ahead of time duplicate
+ * this effort across segment partitions. Once that can be shared across
partitions we can safely
+ * create partitions by default and perhaps refine the slicing approach
implemented in this
+ * method. For this reason segment partitions are currently only created in
tests. Users can call
+ * this method at their own risk.
+ *
+ * @lucene.experimental
+ */
+ public static LeafSlice[] slicesWithPartitions(
Review Comment:
I would think so too. I don't know if we'll want to take the desired amount
of slices as an argument. Using max docs per slice is not flexible enough it
seems. That is kind of why I left this method experimental. I think adding a
boolean to the existing method makes a lot of sense though!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
