jimczi opened a new pull request, #16591: URL: https://github.com/apache/lucene/pull/16591
Backport of #16570 to branch_10x. On main we removed the in-heap `mergingDVUpdates` buffer entirely and always reconstruct the doc-values-update merge carry-over from the source segments on disk. Here that reconstruction is gated on the overlay feature, which is opt-in on 10.x (`maxDocValuesOverlays` defaults to 0), so existing indices keep exactly the behavior they have today. When overlays are enabled, an update that resolves onto a segment while it is being merged is carried over by reading it back from the source segments at merge commit: for each field we collapse the changes flushed since the merge-reader baseline into one packet and add the residual updates that resolved but were not yet flushed. The in-heap buffer is skipped, so we no longer hold a second copy of every resolved numeric or binary value for the whole duration of a merge. When overlays are disabled we keep consuming the buffered packets exactly as before, and the accumulation into that buffer is gated off only on the disk path so there is no wasted memory either way. The trade-off is the same as on main: less heap during merges in exchange for re-reading the updated values from disk at commit time. Correctness is unchanged on the default path since it is untouched. Tests cover both paths. `TestMergeCarryOverFromDisk` deterministically pauses a merge mid-flight, resolves numeric, binary and soft-delete updates onto the merging segments, then releases the merge, running each scenario with overlays off (buffered path) and on (disk path). The existing `TestIncrementalDocValuesUpdates` already drives numeric and binary updates through merges with overlays enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
