jimczi opened a new pull request, #16554: URL: https://github.com/apache/lucene/pull/16554
Backport of #16418 (incremental doc-values updates) to `branch_10x`. Opening this mostly to confirm we're ok backporting it to 10.x, since it touches the segments file format. ### What it does A set-only doc-values update (NUMERIC or BINARY) is written as a sparse "delta" overlay holding just the updated docs, layered over the base column at read time instead of rewriting the whole column. Turns per-update write amplification from O(column) into O(updated docs). Unlike on main, it's **opt-in on 10.x** (`DEFAULT_MAX_DOC_VALUES_OVERLAYS = 0`), since it's a new feature landing in a minor. Enable with `IndexWriterConfig#setMaxDocValuesOverlays(n)`. ### The thing worth confirming for 10.x The segments file format gets a new version, `SegmentInfos.VERSION_10_6`, written unconditionally on every commit (same as main). So any index written by 10.6+ is rejected by older 10.x readers with `IndexFormatTooNewException`, whether or not it uses the feature. I kept the bump unconditional to match main rather than gating it on overlay presence. It follows the forward-only policy (older Lucene never promises to read a newer index) and matches the `VERSION_86` precedent, which was itself introduced in 8.6. Flagging it since a format bump for every index in a minor is the part most worth a second opinion. ### Differences from the main commit - Feature is opt-in (`DEFAULT_MAX_DOC_VALUES_OVERLAYS = 0` instead of 16). - `VERSION_11_0` renamed to `VERSION_10_6` (same value, named for the release it lands in). - `SegmentDocValuesProducer#getSkipper` keeps `throws IOException` here, since `DocValuesProducer#getSkipper` still declares it on 10.x (main dropped it in an unrelated change). - The standalone benchmark is not backported (it will be removed from main separately). Otherwise a straight cherry-pick of 75eddfaff22, plus the follow-up coverage from #16550 (soft deletes over the overlay, addIndexes(Directory) carry-over, and an explicit checkIndex on the deep fold path). Plan is to let it bake on main a bit longer before this merges, so no rush here, just want the backport signed off. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
