fthevenet opened a new issue, #12211: URL: https://github.com/apache/lucene/issues/12211
### Description Hi, I use `DrillSideways` quite heavily in a project of mine and it recently realized that sometimes some documents that *should* match a query do not, whenever at least one component of the `DrillDownQuery` involved was of type `PhraseQuery`. This behaviour would be reproducible **every** time from a freshly started application, provided that the indexed corpus stays _exactly the same_. To make matter worst, however, the document that wouldn't match in a given corpus, would something match when part of a slightly different corpus (i.e. more or less documents), and also, after the application had been running for a while, documents that would not match previously would suddenly start showing up... After quite a bit of debugging, I'm happy to report I believe I have a pretty good idea of what the issue is (and how to fix it). In a nutshell, the problem arises from what I believe is a bug in `DrillSidewaysQueryScorer::score`method, where in the course of a search the following code is called: https://github.com/apache/lucene/blob/0782535017c9e737350e96fb0f53457c7b8ecf03/lucene/facet/src/java/org/apache/lucene/facet/DrillSidewaysScorer.java#L132-L136 First of all, the fact that we call `baseIterator.nextDoc()` here appears inefficient in the event that the iterator for the scorer is a "Two phase iterator", i.e. one that allows to iterate on an approximation of the query matches and confirm that an approximated doc is indeed a match in a second phase, as in this case `baseIterator.nextDoc()` will always call the more costly matcher and never the approximation. But the real problem is the fact that in this case this will lead to a first call to `ExactPhraseMatcher::nextMatch` method and then shortly after a second call to the same matcher as part of the more in depth doc collection: https://github.com/apache/lucene/blob/0782535017c9e737350e96fb0f53457c7b8ecf03/lucene/facet/src/java/org/apache/lucene/facet/DrillSidewaysScorer.java#L148-L157 In either doQueryFirstScoring or doDrillDownAdvanceScoring, a *second* call to `TwoPhaseMatcher::matches` will be made without the iterator has been re-positioned with a call to `TwoPhaseMatcher::nextDoc`, which the documentation explicitly warns against: ``` java /** * Return whether the current doc ID that {@link #approximation()} is on matches. This method * should only be called when the iterator is positioned -- ie. not when {@link * DocIdSetIterator#docID()} is {@code -1} or {@link DocIdSetIterator#NO_MORE_DOCS} -- and at most * once. */ public abstract boolean matches() throws IOException; ``` And indeed what I observed is that although an attempt is made to reset the matcher itself in between these two calls to `TwoPhaseMatcher::matches`, this does not reset *the state within the posting themselves*, and this causes some undefined behaviours when attempting to match as inner positions overflows. Based on these findings, I believe the appropriate fix is to replace: https://github.com/apache/lucene/blob/0782535017c9e737350e96fb0f53457c7b8ecf03/lucene/facet/src/java/org/apache/lucene/facet/DrillSidewaysScorer.java#L133 with: ``` java baseApproximation.nextDoc() ``` I have back-ported that change in a local build of 9.5.0 and confirmed that if fixes my problem; I will therefore submit a pull-request with the hope that it gets reviewed and merged if deemed the correct way to address the issue. Cheers, Frederic ### Version and environment details - OS: Fedora 37 / Windows 10 - Java Runtime: OpenJDK Runtime Environment Temurin-19.0.2+7 (build 19.0.2+7) - Lucene version: 9.5.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
