javanna commented on PR #13542: URL: https://github.com/apache/lucene/pull/13542#issuecomment-2329360505
> I think it's important at least to ensure that we "do no harm" to the existing use cases. Intra-segment is not enabled by default and requires additional work. The scope of this initial support is to set the foundations and make the necessary breaking changes (now listed in the migrate guide). Also the added abstractions are experimental. > do we want to have docid intervals that span segments? There was some discussion above about how the intervals are defined and it wasn't clear whether or not we may have painted ourselves into a corner. Do we have enough flexibility? too much? The technical approach has been the same since this PR was opened: use the existing `BulkScorer#score` method that takes min and max. That is simple and effective. The new experimental abstraction (`LeafReaderContextPartition`) allows to partition a segment. If we do so, each partition must get a different `LeafSlice` (existing concept). A new `slicesWithPartitions` static method is added that is used in tests, and can be entirely customized in how segments are partitioned. You can have multiple partitions in the same slice, but they ought to be from different segments. Does this clarify your concerns? I don't entirely follow how we may have painted ourselves in a corner and how running benchmarks relates to this. I will run benchmarks at some point, but is that necessary to verify that searching force-merged segments can use more of the available resources, same as we do when searching multiple segments concurrently? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
