mkierus opened a new issue, #16702:
URL: https://github.com/apache/lucene/issues/16702
### Description
While testing a migration from Elasticsearch 7.10 to OpenSearch, we noticed
that a query with a `MUST_NOT` clause on a multi-valued numeric field runs
several times slower on OpenSearch 3.x than on earlier versions. We first
noticed this by loading the same dataset into Docker containers running each
engine version and running the same reference query against each one. We traced
this back to Lucene: `IndexOrDocValuesQuery` picks the doc-values side instead
of the points side when the rest of the query is very selective and the
excluded value has many owners. We reproduced this on both a real index and a
synthetic example (attached), and checked it across versions: 9.12.1 is
fine, but 10.1.0, 10.3.2, 10.5.0, and even `main` (11.0.0-SNAPSHOT, built
2026-09-22) are all slow. The data and code behind this are below.
Investigated with AI assistance; every claim here is backed by the attached
reproducer or a direct source quote.
**Source**
Here's the relevant code from `IndexOrDocValuesQuery.java` - it hasn't
changed between 10.1.0 and `main`:
```java
public Scorer get(long leadCost) throws IOException {
final long threshold = cost() >>> 3; // cost() =
indexScorerSupplier.cost(): estimated matches via points
if (threshold <= leadCost) {
return indexScorerSupplier.get(leadCost); // points
} else {
return dvScorerSupplier.get(leadCost); // doc values
}
}
public BulkScorer bulkScorer() throws IOException {
return indexScorerSupplier.bulkScorer(); // always points
}
```
The catch is that `BooleanScorerSupplier` fetches `MUST_NOT` clauses through
`get(leadCost)`, not through `bulkScorer()`:
```java
for (ScorerSupplier ss : subs.get(Occur.MUST_NOT)) {
prohibited.add(ss.get(positiveScorerCost));
}
```
So if the rest of the query is very selective (a low `leadCost`) and the
excluded value has a lot of owners (a high `cost()`), you end up on the
doc-values side. Before #13408 (Lucene 10.0.0), the prohibited clause was built
with `w.scorer(context)` instead, which has no cost bound, so it always went
through the points side - which is why 9.12.1 isn't affected.
**Reproducer 1: real index**
We took a real OpenSearch 3.0.0 shard and opened it directly with plain
Lucene 10.1.0, bypassing OpenSearch. It has about 409,845 documents, and a
handful of small segments are made up almost entirely of documents carrying
~470,000 values in one multi-valued integer field. The query is a term match on
another field (matching 6 documents) as `MUST`, plus a `MUST_NOT` on that big
field.
| excluded value | owns the fat docs | top-1 (sort by doc) |
|---|---|---:|
| real user id | yes, nearly all | 316-675 ms |
| 3 other ids | no | 0.5-5 ms |
| points-only `MUST_NOT` | any value | 2-6 ms |
Here's what `hot_threads` shows while one of the slow queries is running:
```
Lucene90DocValuesProducer$VaryingBPVReader.getLongValue
<- Lucene90DocValuesProducer$23.nextValue
<- SortedNumericDocValuesRangeQuery$1$2.matches
<- ReqExclBulkScorer.score (line 60)
```
Breaking the timing down per segment, the cost sits entirely in the segments
made of those fat documents, and it scales with how many values each segment
holds - roughly 3 ns per value.
**Reproducer 2: synthetic (`Repro.java` attached)**
We built an index of N documents, where the last F of them are "fat" - about
50,000 values each, indexed as both `IntPoint` and
`SortedNumericDocValuesField` - and every one of those fat documents contains
the value we're excluding. The query is 3 required terms plus a term that
matches 3 documents, with a `MUST_NOT IndexOrDocValuesQuery` on the excluded
value. We ran it with the matching group sitting at the start, the middle, and
the end of the fat block.
With 200,000 documents, 500 of them fat at 50,000 values each, one segment,
median of 7 runs:
| required doc at | 9.12.1 | 10.1.0 | 10.3.2 | 10.5.0 | main |
|---|---:|---:|---:|---:|---:|
| start of fat block | 0.2 ms | 9.1 ms | 9.7 ms | 9.2 ms | 12.4 ms |
| middle | 0.1 ms | 4.7 ms | 4.8 ms | 4.9 ms | 6.5 ms |
| end (no run follows) | 0.1 ms | 0.2 ms | 0.2 ms | 0.1 ms | 0.1 ms |
| points-only `MUST_NOT` | ~0.2 ms | 0.2-0.3 ms | 0.3 ms | 0.2-0.4 ms |
0.2-0.3 ms |
The cost tracks how many excluded documents follow the required one. We also
tried scoring in 4096-document windows, the way OpenSearch's
`CancellableBulkScorer` does it, by driving
`Weight#bulkScorer`/`ScorerSupplier#bulkScorer` directly - no OpenSearch
involved - and it made no difference.
**Workarounds**
- Mapping the field as `keyword` instead of numeric moves the exclusion onto
postings: 196-326 ms on the real-world query above, versus tens of seconds with
the numeric mapping.
- Turning off doc values (`doc_values: false`) is fast too, but it breaks
any script or aggregation that reads the field.
- Or just use a points-only query for the `MUST_NOT` clause and skip
`IndexOrDocValuesQuery` altogether.
**Attachments**
-
[Repro.java](https://github.com/user-attachments/files/32518126/Repro.java) -
the code for reproducer 2, self-contained, just needs a `lucene-core` jar.
-
[lucene-repro.zip](https://github.com/user-attachments/files/32518112/lucene-repro.zip)
- a small Docker harness (`Dockerfile` + `run.sh`) that runs `Repro.java`
against several Lucene versions in one go, including building `main` from
source. This is what produced the table above.
### Version and environment details
Lucene versions: 9.12.1, 10.1.0, 10.3.2, 10.5.0, main (11.0.0-SNAPSHOT,
shallow-cloned and built 2026-09-22)
Java: OpenJDK 21.0.12 (2026-07-21 LTS) — used to build and run the
reproducer for all of the above
Host: macOS, Docker containers on linux/arm64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
