hossman opened a new issue, #16573:
URL: https://github.com/apache/lucene/issues/16573
### Description
When attempting to upgrade Solr from Lucene 10.4.0 -> 10.5.1 the Solr tests
identified a regression causing test failures in (single-valued) float & double
based fields that use both points & docvalues: Range queries are failing to
match all expected documents.
* Identical Solr tests using multivalued fields still pass.
* Identical Solr tests using _only_ docvalues, or _only_ points also still
pass.
* Similar Solr tests using int & long types still pass regardless of
multivalue/points/dv permutations
After writing a scaled down Lucene equivilent test, git bisect identified
`da18ffc5890f72df279373d40ce5c4192734cd11` (aka: #15760) as the cause of the
bug.
I'm not an expert here, but the underlying issue seems to be that #15760
modified `SortedNumericDocValuesRangeQuery.rewrite` to refactor some logic
identifying min/max values "on disk" (in order to optimize away queries known
to be out of range of any valid docs) into a new helper function
(`NumericFieldStats.getStats`) that not only considers the stats of the
docvalues (if they exist), but (starting with #15760) this method also (first)
looks at the stats from any `PointValues` -- and those stats may differ from
what's _actually_ in the doc values.
The problem being that there is no guarantee that a `PointsField` named
`foo` has the same internal `long` values in it as a `DocValues` field named
`foo`.
IIUC, fields like `DoublePoint` and `FloatPoint` internally use the same
encodings methods in `NumericUtils` that are recommended when using
`SortedNumericDocValuesField` -- but AFAIK there is no requirement that all
doubles & floats be encoded that way when using `NumericDocValuesField` --
which is also supported by `SortedNumericDocValuesRangeQuery`.
* Solr uses `Double.doubleToLongBits` with `NumericDocValuesField` for
historic reasons (predating Lucene's `NumericUtils` IIRC?)
* likewise for Floats
* Lucene's `DoubleDocValuesField` helper class uses
`Double.doubleToRawLongBits`
* likewise for Floats
* Neither seem to safely work with this new `NumericFieldStats.getStats()`
logic
I'm attaching a quickly hacked together patch with a test case demonstrating
the problem:
*
[lucene_bug_caused_by_github15760.patch](https://github.com/user-attachments/files/31533966/lucene_bug_caused_by_github15760.patch).
This tests passes with Lucene 10.4.0
I suspect the only viable fix here is to remove the `PointValues`
optimizations added in #15760 ... or ... _maybe_: make the `PointValues`
optimization used if-and-only-if we can confirm from the IndexReader that
`SortedNumericDocValues` are in use -- since `SortedNumericDocValues` they are
documented as requiring the use of the sae `NumericUtils` encoding for double &
float that Points use (IIUC this is easy/cheap to do? ... just check if `null
!= LeafReader.getSortedNumericDocValues()` ... correct?)
### Version and environment details
Lucene 10.5.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
