[ 
https://issues.apache.org/jira/browse/SOLR-14524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson reopened SOLR-14524:
-----------------------------------

Saw this one go by today, so reopening...

 

{color:#000000}Build: 
{color}[https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/26913/]
{color:#000000}Java: 64bit/jdk-11.0.6 -XX:-UseCompressedOops 
-XX:+UseConcMarkSweepGC{color}

{color:#000000}1 tests failed.{color}
{color:#000000}FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test{color}

{color:#000000}Error Message:{color}
{color:#000000}Long running task finished too early, can't test expected null, 
but was:<1591494388303>{color}

{color:#000000}Stack Trace:{color}
{color:#000000}java.lang.AssertionError: Long running task finished too early, 
can't test expected null, but was:<1591494388303>{color}
 {color:#000000}at 
__randomizedtesting.SeedInfo.seed([F72E290521C3A7BB:7F7A16DF8F3FCA43]:0){color}
 {color:#000000}at org.junit.Assert.fail(Assert.java:88){color}
 {color:#000000}at org.junit.Assert.failNotNull(Assert.java:755){color}
 {color:#000000}at org.junit.Assert.assertNull(Assert.java:737){color}
 {color:#000000}at 
org.apache.solr.cloud.MultiThreadedOCPTest.testFillWorkQueue(MultiThreadedOCPTest.java:94){color}
 {color:#000000}at 
org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:67){color}
 {color:#000000}at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method){color}
 {color:#000000}at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62){color}
 {color:#000000}at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){color}
 {color:#000000}at 
java.base/java.lang.reflect.Method.invoke(Method.java:566){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992){color}
 {color:#000000}at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1081){color}
 {color:#000000}at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1053){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57){color}
 {color:#000000}at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49){color}
 {color:#000000}at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45){color}
 {color:#000000}at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48){color}
 {color:#000000}at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64){color}
 {color:#000000}at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57){color}
 {color:#000000}at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36){color}
 {color:#000000}at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36){color}
 {color:#000000}at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53){color}
 {color:#000000}at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47){color}
 {color:#000000}at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64){color}
 {color:#000000}at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370){color}
 {color:#000000}at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826){color}
 {color:#000000}at java.base/java.lang.Thread.run(Thread.java:834){color}

> Harden MultiThreadedOCPTest
> ---------------------------
>
>                 Key: SOLR-14524
>                 URL: https://issues.apache.org/jira/browse/SOLR-14524
>             Project: Solr
>          Issue Type: Test
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: SolrCloud
>    Affects Versions: master (9.0)
>            Reporter: Ilan Ginzburg
>            Assignee: Mike Drob
>            Priority: Minor
>              Labels: test
>             Fix For: master (9.0)
>
>          Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> {{MultiThreadedOCPTest.test()}} fails occasionally in Jenkins because of 
> timing of tasks enqueue to the Collection API queue.
> This test in {{testFillWorkQueue()}} enqueues a large number of tasks (115, 
> more than the 100 Collection API parallel executors) to the Collection API 
> queue for a collection COLL_A, then observes a short delay and enqueues a 
> task for another collection COLL_B.
>  It verifies that the COLL_B task (that does not require the same lock as the 
> COLL_A tasks) completes before the third COLL_A task.
> Test failures happen because when enqueues are slowed down enough, the first 
> 3 tasks on COLL_A complete even before the COLL_B task gets enqueued!
> In one sample failed Jenkins test execution, the COLL_B task enqueue happened 
> 1275ms after the enqueue of the first COLL_A, leaving plenty of time for a 
> few (and possibly all) COLL_A tasks to complete.
> Fix will be along the lines of:
>  * Make the “blocking” COLL_A task longer to execute (currently 1 second) to 
> compensate for slow enqueues.
>  * Verify the COLL_B task (a 1ms task) finishes before the long running 
> COLL_A task does. This would be a good indication that even though the 
> collection queue was filled with tasks waiting for a busy lock, a non 
> competing task was picked and executed right away.
>  * Delay the enqueue of the COLL_B task to the end of processing of the first 
> COLL_A task. This would guarantee that COLL_B is enqueued once at least some 
> COLL_A tasks started processing at the Overseer. Possibly also verify that 
> the long running task of COLL_A didn't finish execution yet when the COLL_B 
> task is enqueued...
>  * It might be possible to set a (very) long duration for the slow task of 
> COLL_A (to be less vulnerable to execution delays) without requiring the test 
> to wait for that task to complete, but only wait for the COLL_B task to 
> complete (so the test doesn't run for too long).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to