zihanx opened a new issue, #15905:
URL: https://github.com/apache/lucene/issues/15905

   ## Description:
   
   Following up on the recently merged ReaderUtil.partitionByLeaf API 
(https://github.com/apache/lucene/pull/15507), we're planning a second task to 
build a generic utility class for efficiently retrieving doc-values for an 
int[] of global doc IDs.
   
   The doc-values retriever would:
   
   - Take an `IndexReader`, `int[] globalDocIds`, and `DoubleValuesSource[]` 
(or abstract LeafVisitor factory)
   
   - Take an Executor/TaskExecutor for concurrent per-leaf retrieval
   
   - Return values in 1:1 correspondence with the input doc ID order
   
   This requires a scatter/gather pattern:
   
   - Scatter: Partition global doc IDs by leaf (using `partitionByLeaf`)
   
   - Map: Retrieve doc-values per leaf (concurrent)
   
   - Gather: Reassemble results back to original input order
   
   The current `partitionByLeaf(ScoreDoc[], leaves)` API doesn't track original 
positions, so we can't reassemble results in input order. We need ordinal 
tracking.
   
   ## Design options:
   
   1. Two separate APIs (current prototype):
   ```
   // Existing - no ordinals (for callers who don't need gather and only take 
in ScoreDoc[])
   public static int[][] partitionByLeaf(ScoreDoc[] hits, 
List<LeafReaderContext> leaves)
   
   // New - with ordinals for scatter/gather
   public record PartitionedHits(int[][] docIdsByLeaf, int[] ordinals) {}
   public static PartitionedHits partitionByLeafWithOrdinals(int[] 
globalDocIds, List<LeafReaderContext> leaves)
   
   // Private - helper method for duplicate partition logic 
   private static int[][] partitionSortedDocIds(int[] sortedDocIds, 
List<LeafReaderContext> leaves) 
   ```
   Pros: Clear intent, no wasted compute, type-safe
   
   Cons: Two entry points
   
   2. Single API with boolean flag:
   ```
   public record PartitionedHits(int[][] docIdsByLeaf, int[] ordinals) {}
   public static PartitionedHits partitionByLeaf(..., boolean includeOrdinals)
   ```
   Pros: Single entry point
   
   Cons: ordinals is null when not requested,  still need to handle different 
input types (ScoreDoc[] vs int[])
   
   3. Always return ordinals:
   ```
   public static PartitionedHits partitionByLeaf(int[] globalDocIds, 
List<LeafReaderContext> leaves)
   ```
   Pros: Simplest API
   
   Cons: Wastes compute for callers who don't need ordinals
   
   ## Questions for the community:
   
   1. Should we add an `int[]` input method? The merged PR only has 
`ScoreDoc[]` input 
      ([last PR 
comment](https://github.com/apache/lucene/pull/15803#discussion_r2960596460)), 
but the scatter/gather use case needs raw doc IDs.
   
   2. Which API design is preferred for ordinal tracking? Or any better ideas?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to