Re: Cassandra DTests deadlocks on random test

2017-10-21 Thread Lutaya Shafiq Holmes
; Sent: Friday, October 20, 2017 3:28 PM >> To: dev@cassandra.apache.org >> Subject: Re: Cassandra DTests deadlocks on random test >> >>> On 10/20/2017 02:30 AM, Sergey wrote: >>> >>> I believe the problem can be not related to RAM space, since today I >

Re: Cassandra DTests deadlocks on random test

2017-10-21 Thread Jeff Jirsa
20, 2017, at 10:17 PM, Sergey wrote: > > What environment are you guys using to run dtests? > > Best regards, > Sergey > > From: Michael Shuler > Sent: Friday, October 20, 2017 3:28 PM > To: dev@cassandra.apache.org > Subject: Re: Cassandra DTests deadlocks on random

RE: Cassandra DTests deadlocks on random test

2017-10-20 Thread Sergey
What environment are you guys using to run dtests? Best regards, Sergey From: Michael Shuler Sent: Friday, October 20, 2017 3:28 PM To: dev@cassandra.apache.org Subject: Re: Cassandra DTests deadlocks on random test On 10/20/2017 02:30 AM, Sergey wrote: > > I believe the problem can

Re: Cassandra DTests deadlocks on random test

2017-10-20 Thread Michael Shuler
On 10/20/2017 02:30 AM, Sergey wrote: > > I believe the problem can be not related to RAM space, since today I > run into the same issue on 200GB RAM machine. Awesome! You've found a bug with either cassandra-dtest or cassandra. Welcome to Apache Cassandra Test Engineering. :) -- Kind regards,

RE: Cassandra DTests deadlocks on random test

2017-10-20 Thread Sergey
@cassandra.apache.org Subject: Re: Cassandra DTests deadlocks on random test 7.6G RAM may be a little bit too small, we've seen similar random hangs in the past on non-resource-intensive tests on m3.large. It doesn't appear you are skipping resource-intensive tests. Our standard dtest instance type has

Re: Cassandra DTests deadlocks on random test

2017-10-19 Thread Murukesh Mohanan
I have had this problem too on similarly spec'd OpenStack instances (but I'm reasonably certain I didn't include the resource-intensive tests). My solution was to run the dtests in small batches (say 5-10 each), with a timeout (say 1.2x the max for 5 tests from a good run). Kill the test if exceeds

Re: Cassandra DTests deadlocks on random test

2017-10-19 Thread Michael Shuler
7.6G RAM may be a little bit too small, we've seen similar random hangs in the past on non-resource-intensive tests on m3.large. It doesn't appear you are skipping resource-intensive tests. Our standard dtest instance type has been an m3.xlarge, and the resource-intensive tests are run m3.2xlarge.