krickert commented on PR #15800: URL: https://github.com/apache/lucene/pull/15800#issuecomment-4020453463
You're right @benwtrent. Didn't think of that and as you suggested, my new test shows that in all scenarios, at least one document was missed. My goal isn't optimization though. I want a collector that targets document matches, not $K$ chunks. The interface I had in mind and updated (I removed the pruning): "Give me the top $D$ documents, but cap the search at $K$ visits." Right now you'd have to guess a large $K$ and hope it covers enough unique documents. The harder problem is large documents - a single doc's chunks can dominate the heap and crowd out better matches, so the number of distinct documents returned is never guaranteed. The collector manages a heap of $D$ unique documents and uses visitedLimit as a hard ceiling, similar to how $K$ works today. The standard search path is unchanged. Co-location matters for MaxSim consistency. Same shard gives you a clean top-chunks-per-doc view; distributed falls back to standard KNN with a more meaningful termination signal at the shard level. Once stable, this opens a post-ranking step: return the top $N$ chunks within those $D$ documents, which enables chunk-level highlighting across a document without a second search pass. Useful for RAG to know you get back a specific number of documents. Results against sentences-1024d (73k docs, 1-vector-per-doc): Metric | DistinctDocs (New) | Standard HNSW (Baseline) -- | -- | -- Recall | 0.996 | 0.996 Latency (ms) | 1.447 | 1.336 Net CPU | 1.445 | 1.334 Recall is identical. The ~8% overhead on latency and CPU is the per-document bookkeeping cost - I suspect there's room to bring that down, but wanted to get the baseline numbers in front of you first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
