Hi Michael,
I believe the problem can be not related to RAM space, since today I run into
the same issue on 200GB RAM machine.
I will try to run intensive/non-intensive tests separately.
Thank you,
Sergey
From: Michael Shuler
Sent: Thursday, October 19, 2017 3:58 PM
To: dev@cassandra.apache.org
Subject: Re: Cassandra DTests deadlocks on random test
7.6G RAM may be a little bit too small, we've seen similar random hangs
in the past on non-resource-intensive tests on m3.large. It doesn't
appear you are skipping resource-intensive tests. Our standard dtest
instance type has been an m3.xlarge, and the resource-intensive tests
are run m3.2xlarge. (or something comparable on RAM & SSD)
dtest run command (excludes resource-intensive tests):
https://github.com/apache/cassandra-builds/blob/master/build-scripts/cassandra-dtest.sh#L53
dtest-large run command (only resource-intensive tests):
https://github.com/apache/cassandra-builds/blob/master/build-scripts/cassandra-dtest.sh#L61
Running dtest without excluding resource-intensive will run everything.
If you wish to troubleshoot your tests when they hang, there should be
/tmp/dtest-X directories with the ccm cluster left on disk from the
hung test, since they never get to cleanup stage.
--
Michael
On 10/19/2017 05:29 AM, Sergey La wrote:
> Hi!
>
> I have created the patch for the Cassandra version 3.0.14 and trying to
> test it using the cassandra dtests.
>
> Problem is - dtests deadlocks at some random tests, time and again - on
> unpatched 3.0.14 version of Cassandra.
>
> What I have done.
>
> I cloned the cassandra repository (origin
> http://git-wip-us.apache.org/repos/asf/cassandra.git), and checked out to
> tags/cassandra-3.0.14 - head is on f3e38cb638113c2a23855a104d6082da5bc10ddb.
>
> Then I have cloned the cassandra-dtest repo (origin git://
> github.com/riptano/cassandra-dtest.git). Head is on
> 6843d76d0a85ad82edf889e8280b87786dc48486.
>
> I setup dtests according to this instructions:
> https://github.com/riptano/cassandra-dtest/blob/master/INSTALL.md
>
> In addition, I have setup JAVA8_HOME and JAVA_HOME variables to public jre
> of my 1.8.0_144 jdk.
>
> I start testing using this command:
> JAVA8_HOME=$JAVA8_HOME nosetests --with-flaky --with-xunit
> --xunit-file=out.xunit.xml --force-flaky --max-runs=3 --verbose
> --debug-log=err.debug.nose.txt 1> out.txt 2> err.txt
> I run the tests on x86_64 CentOS 7 with 7.6G of RAM.
>
> Problem symptoms:
> During the "normal" run (I have got only 1 "normal" run in 5 attempts),
> err.txt is updated constantly with name of the test recently completed, and
> in the end out.xunit.xml file appears, with test summary results. nosetests
> process exits.
>
> During the "problem" run tests stop progressing (err.txt was not modified
> for 10 hours), out.xunit.xml is not appearing, nosetests process runs. I
> killed java processes, but nothing changed for 2 hours - nosetests process
> still runs, but files are unchanged.
>
> Any help would be appreciated,
> Sergey
>
-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org
