[
https://issues.apache.org/jira/browse/LUCENE-10093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427239#comment-17427239
]
Michael McCandless commented on LUCENE-10093:
---------------------------------------------
Yeah I agree with [~jpountz]'s explanation. And, the test randomized itself
into a nice corner-case of asking force merge to reduce an unusually large
number ({{atLeast(100)}} --> {{408}}) of {{0.1 MB}} segments down to an
unusually small number of segments ({{random().nextInt(10) + 3}} --> {{3}}) in
one go, while also asking for the (over-constrained) {{maxMergedSegmentMB=10}}.
I think we should relax the {{assert}} to agree with the current
implementation, allowing {{maxSegmentCount}} to win over {{maxMergedSegmentMB}}
in general when they conflict. I'll take a stab.
I will also update {{TieredMergePolicy}}'s javadocs to advertise this tricky
conflicting situation. Hmm, except, the javadocs [currently seem to state the
opposite|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java#L54-L63]
(i.e. that {{maxSegmentCount > 1}} is passed to {{findForcedMerges}} and it
conflicts with {{maxMergedSegmentMB * 1.25}}, then the latter wins,
confusing!). I think that may have been added from LUCENE-8370 (edit:
[yes|https://github.com/apache/lucene-solr/commit/1f5c75c]). But it really
looks like the code is doing the opposite of these recently added best effort
javadocs. I will study some more to convince myself :)
In general if you {{forceMerge}} your index, and then continue indexing, you
can get it into "unnatural" ghost- or zombie-like states, which I think was the
root cause initiating LUCENE-7976. It's a little spooky just how such an index
would then naturally grow/merge as more docs are aggressively indexed into it.
Baby ghost/zombie steps! We do need to think of a better long-term solution
for TMP to gracefully deal with the ghosts of {{forceMerge}} past.
> TestTieredMergePolicy.testForcedMergesUseLeastNumberOfMerges test failure
> -------------------------------------------------------------------------
>
> Key: LUCENE-10093
> URL: https://issues.apache.org/jira/browse/LUCENE-10093
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael McCandless
> Priority: Major
>
> This test fails periodically in our CI builds, and the failing seed repros
> for me:
> {noformat}
> org.apache.lucene.index.TestTieredMergePolicy > test suite's output saved to
> /l/trunk/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestTieredMergePolicy.txt,
> copied below:
> > java.lang.AssertionError
> > at
> __randomizedtesting.SeedInfo.seed([7B591E657503510C:C958DC291BD5CF0A]:0)
> > at org.junit.Assert.fail(Assert.java:87)
> > at org.junit.Assert.assertTrue(Assert.java:42)
> > at org.junit.Assert.assertTrue(Assert.java:53)
> > at
> org.apache.lucene.index.TestTieredMergePolicy.assertMaxSize(TestTieredMergePolicy.java:497)
> > at
> org.apache.lucene.index.TestTieredMergePolicy.testForcedMergesUseLeastNumberOfMerges(TestTieredMergePolicy.java:454)
> > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
> > at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:567)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
> > at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
> > at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> > at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
> > at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> > at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
> > at java.base/java.lang.Thread.run(Thread.java:831)
> 2> NOTE: reproduce with: gradlew test --tests
> TestTieredMergePolicy.testForcedMergesUseLeastNumberOfMerges
> -Dtests.seed=7B591E657503510C -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=es-GT -Dtests.timezone=Europe/Istan\
> bul -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> 2> NOTE: test params are: codec=Asserting(Lucene90): {}, docValues:{},
> maxPointsInLeafNode=467, maxMBSortInHeap=5.280570030610025,
> sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=es-GT,
> timezone=Europe/Istanbul
> 2> NOTE: Linux 5.13.12-arch1-1 amd64/N/A 16.0.2
> (64-bit)/cpus=128,threads=1,free=259208456,total=268435456
> 2> NOTE: All tests run in this JVM: [TestTieredMergePolicy] {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
