Adrien Grand created LUCENE-10073:
-------------------------------------
Summary: Allow very small merges to merge more than
segmentsPerTier segments?
Key: LUCENE-10073
URL: https://issues.apache.org/jira/browse/LUCENE-10073
Project: Lucene - Core
Issue Type: Improvement
Reporter: Adrien Grand
If you are doing lots of concurrent indexing, NRT search regularly publishes
many tiny segments, which in-turn pushes a lot of pressure on merging, which
needs to constantly merge these tiny segments so that the total number of
segments of the index remains under the budget.
In parallel, TieredMergePolicy's behavior is to merge aggressively segments
that are below the floor size. The budget of the number of segments allowed in
the index is computed as if all segments were larger than the floor size, and
merges that only contain segments below the floor size get a perfect skew which
guarantees them to get a better score than any merge that contains one or more
segments above the floor size.
I'm considering reducing the merging overhead in the NRT case by raising
maxMergeAtOnce and allowing merges to merge more than mergeFactor segments as
long as the number of merged segments is below maxMergeAtOnce and the merged
segment size is below the floor segment size.
Said otherwise, "normal" merges would be allowed to merge up to mergeFactor
segments like today, while small merges (size of the merged segment < floor
segment bytes) could go up to maxMergeAtOnce.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
