javanna commented on code in PR #13542:
URL: https://github.com/apache/lucene/pull/13542#discussion_r1750743766
##########
lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java:
##########
@@ -363,7 +374,88 @@ public static LeafSlice[] slices(
LeafSlice[] slices = new LeafSlice[groupedLeaves.size()];
int upto = 0;
for (List<LeafReaderContext> currentLeaf : groupedLeaves) {
- slices[upto] = new LeafSlice(currentLeaf);
+ slices[upto] =
+ new LeafSlice(
+ new ArrayList<>(
+ currentLeaf.stream()
+ .map(LeafReaderContextPartition::createForEntireSegment)
+ .toList()));
+ ++upto;
+ }
+
+ return slices;
+ }
+
+ /**
+ * Creates leaf slices that leverage intra-segment concurrency by splitting
segments into multiple
+ * partitions according to the provided max number of documents per slice
and maximum number of
+ * segments per slice. If a segment holds more documents than the provided
max per slice, it gets
+ * split into equal size partitions that each gets its own slice assigned.
+ *
+ * <p>Note: this method is not yet called by the default slicing
implementation {@link
+ * #slices(List)}. Certain queries that require segment-level computation
ahead of time duplicate
+ * this effort across segment partitions. Once that can be shared across
partitions we can safely
+ * create partitions by default and perhaps refine the slicing approach
implemented in this
+ * method. For this reason segment partitions are currently only created in
tests. Users can call
+ * this method at their own risk.
+ *
+ * @lucene.experimental
+ */
+ public static LeafSlice[] slicesWithPartitions(
Review Comment:
Looks like the existing IndexSearcher#slices (that public static one) does
not have a lucene.experimental tag. That's odd, because the rest of the search
concurrency code is marked experimental. On the other hand, it should be fine
adding a last argument boolean to the existing method, that's the direction I
am taking. I added that to the MIGRATE file too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
