jimczi opened a new pull request, #16847:
URL: https://github.com/apache/lucene/pull/16847
**Draft — opening this to show the approach and get early feedback, not as a
finished change.**
This is an attempt at letting a single index hold more than
`Integer.MAX_VALUE - 128`
documents. Today a doc is an `int` everywhere, so once the segments together
pass ~2.1B
docs there's nowhere to put the number and you have to shard. I wanted to
see how far you
get by widening only the whole-index doc id to `long` and leaving everything
inside a
segment alone.
The idea the whole branch rests on: a doc id inside a segment stays an `int`
in
`[0, maxDoc)`, and a segment still can't exceed `MAX_DOCS`. Only the
*composite* doc id —
`leaf.docBase + leafDocId` — becomes a `long`. That line falls between the
leaf and the
reader above it, so codecs and every `*Format` never see a `long` and
there's no file
format change; existing indices read back unchanged, and an index under 2^31
docs pays
nothing.
What that meant in practice:
- `IndexReader` gets `long totalMaxDoc()` / `totalNumDocs()`. The old `int
maxDoc()` /
`numDocs()` still work below 2^31 but now throw `ArithmeticException` once
the real count
doesn't fit, instead of quietly overflowing. Leaf readers keep the `int`
versions.
- `LeafReaderContext.docBase` is a `long`, with one narrowing helper
(`toLeafDocId(long)`)
that asserts the doc really belongs to the leaf, so the `long`→`int` cast
only happens in
one place.
- `ScoreDoc.doc` and `FieldDoc` carry a `long`, and that flows through
`HitQueue`,
`TopDocs`/`TopFieldDocs`, `searchAfter(...)` and `explain(...)`. This is
the part that
actually breaks callers: anything reading `scoreDoc.doc` into an `int`, or
implementing a
`Collector` / `FieldComparator`, has to recompile and retype. It's the
main reason this
can't be backported.
- There are two limits now. The per-segment one (`MAX_DOCS`, `int`) is
unchanged. A new
whole-index one (`MAX_TOTAL_DOCS`, currently `2^37` ≈ 137B) caps the
total; I picked a
bounded number rather than `Long.MAX_VALUE` mostly so merge/accounting
stay sane — happy
to be argued out of the exact value, it's deliberately a round-ish
placeholder.
- The top-scores heap keeps the existing packed `(score, doc)`-in-a-`long`
implementation
for indices up to 2^31 so the common case doesn't regress, and switches to
a `long`-doc
heap only beyond that.
- The composite "treat the whole reader as one leaf" helpers (`MultiTerms`,
`MultiBits`,
`MultiDocValues`) can't represent a `> 2^31` space in their int APIs, so
they now fail
fast with a "consume the index per-leaf instead" message rather than
overflowing.
Testing is awkward because you obviously can't index billions of docs in a
unit test, so
`TestSearchBeyondMaxInt` fakes it: a couple of real one-doc segments are
wrapped in
`FilterLeafReader`s that report an inflated `maxDoc` and placed ahead of the
real leaf in a
`MultiReader`, which pushes that leaf's `docBase` — and its hits' doc ids —
past 2^31 with
no large allocation. From there it checks that hits come back as genuine
`long`s, that
global-doc dispatch (`subIndex`, `storedFields.document(long)`) lands on the
right leaf,
that `searchAfter` paging works across the boundary for score / field /
`_doc` sort, and
that the fail-fast helpers throw the clear message. The rest of the core
suite and every
touched module pass as-is, and I turned `CollectionIncompatibleType` into a
build error to
catch the `Set<Integer>.contains(longDoc)` autobox traps the widening
invites.
Rough edges I already know about, and why it's still a draft:
- A few structures are still sized to the whole index — grouping's
group-heads keep a dense
per-leaf `FixedBitSet`, which is `O(totalMaxDoc)` memory. Fine
functionally, wants a
sparse form before anyone runs this near the top of the range.
- `ReaderUtil.partitionByLeaf` is still `int`-bounded (nothing calls it past
2^31 yet).
- A handful of peripheral spots (some `suggest`, `spatial-extras`,
`monitor`) still throw a
generic `ArithmeticException` past 2^31 instead of the nice per-leaf
message. They fail
safely; I just haven't routed them through the shared guard.
Mostly I'd like to know whether the leaf-int / global-long split is the
right shape before
polishing any of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
