krickert commented on PR #15800:
URL: https://github.com/apache/lucene/pull/15800#issuecomment-4020453463

   You're right @benwtrent. Didn't think of that and as you suggested, my new 
test shows that in all scenarios, at least one document was missed.
   
   My goal isn't optimization though. I want a collector that targets document 
matches, not $K$ chunks. 
   
   The interface I had in mind and updated (I removed the pruning): 
   
   "Give me the top $D$ documents, but cap the search at $K$ visits." 
   
   Right now you'd have to guess a large $K$ and hope it covers enough unique 
documents. The harder problem is large documents - a single doc's chunks can 
dominate the heap and crowd out better matches, so the number of distinct 
documents returned is never guaranteed.
   
   The collector manages a heap of $D$ unique documents and uses visitedLimit 
as a hard ceiling, similar to how $K$ works today. The standard search path is 
unchanged.
   
   Co-location matters for MaxSim consistency. Same shard gives you a clean 
top-chunks-per-doc view; distributed falls back to standard KNN with a more 
meaningful termination signal at the shard level.
   
   Once stable, this opens a post-ranking step: return the top $N$ chunks 
within those $D$ documents, which enables chunk-level highlighting across a 
document without a second search pass. 
   
   Useful for RAG to know you get back a specific number of documents.
   
   Results against sentences-1024d (73k docs, 1-vector-per-doc):
   
   Metric | DistinctDocs (New) | Standard HNSW (Baseline)
   -- | -- | --
   Recall | 0.996 | 0.996
   Latency (ms) | 1.447 | 1.336
   Net CPU | 1.445 | 1.334
   
   Recall is identical. The ~8% overhead on latency and CPU is the per-document 
bookkeeping cost - I suspect there's room to bring that down, but wanted to get 
the baseline numbers in front of you first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to