Re: RE: batch_mutate failed: out of sequence response

2011-04-18 Thread Dan Washusen
Sure is: scale7-pelops-1.1-0.7.x-20110418.225848-3.jar Cheers, -- Dan Washusen On Tuesday, 19 April 2011 at 3:58 PM, Héctor Izquierdo Seliva wrote: > Thanks Dan for fixing that! Is the change integrated in the latest maven > snapshot? > > El mar, 19-04-2011 a las 10:48 +1000, Dan Washusen escri

Re: RE: batch_mutate failed: out of sequence response

2011-04-18 Thread Héctor Izquierdo Seliva
Thanks Dan for fixing that! Is the change integrated in the latest maven snapshot? El mar, 19-04-2011 a las 10:48 +1000, Dan Washusen escribió: > An example scenario (that is now fixed in Pelops): > 1. Attempt to write a column with a null value > 2. Cassandra throws a TProtocolException

Re: RE: batch_mutate failed: out of sequence response

2011-04-18 Thread Dan Washusen
An example scenario (that is now fixed in Pelops): Attempt to write a column with a null value Cassandra throws a TProtocolException which renders the connection useless for future operations Pelops returns the corrupt connection to the pool A second read operation is attempted with the corrupt co

Re: RE: batch_mutate failed: out of sequence response

2011-04-18 Thread Jonathan Ellis
Any idea what's causing the original TPE? On Mon, Apr 18, 2011 at 6:22 PM, Dan Washusen wrote: > It turns out that once a TProtocolException is thrown from Cassandra the > connection is useless for future operations. Pelops was closing connections > when it detected TimedOutException, TTransportE

Re: RE: batch_mutate failed: out of sequence response

2011-04-18 Thread Dan Washusen
It turns out that once a TProtocolException is thrown from Cassandra the connection is useless for future operations. Pelops was closing connections when it detected TimedOutException, TTransportException and UnavailableException but not TProtocolException. We have now changed Pelops to close c

Re: RE: batch_mutate failed: out of sequence response

2011-04-14 Thread Dan Washusen
I've looked over the Pelops code again and I really can't see how it could be at fault here... -- Dan Washusen On Wednesday, 13 April 2011 at 3:20 AM, Stephen McKamey wrote: > [I wrote this Apr 10, 2011 at 12:09 but my message seems to have gotten lost > along the way.] > > I use Pelops (the

RE: batch_mutate failed: out of sequence response

2011-04-12 Thread Stephen McKamey
[I wrote this Apr 10, 2011 at 12:09 but my message seems to have gotten lost along the way.] I use Pelops (the 1.0-0.7.x build from the Github Maven repo) and have occasionally seen this message (under load or during GC). I have a test app running in two separate single-threaded processes doing a

Re: RE: batch_mutate failed: out of sequence response

2011-04-07 Thread Dan Washusen
Pelops uses a single connection per operation from a pool that is backed by Apache Commons Pool (assuming you're using Cassandra 0.7). I'm not saying it's perfect but it's NOT sharing a connection over multiple threads. Dan Hendry mentioned that he sees these errors. Is he also using Pelops? Fro

Re: RE: batch_mutate failed: out of sequence response

2011-04-07 Thread Héctor Izquierdo Seliva
El mié, 06-04-2011 a las 21:04 -0500, Jonathan Ellis escribió: > "out of sequence response" is thrift's way of saying "I got a response > for request Y when I expected request X." > > my money is on using a single connection from multiple threads. don't do > that. > I'm not using thrift direct

Re: RE: batch_mutate failed: out of sequence response

2011-04-07 Thread Héctor Izquierdo Seliva
El mié, 06-04-2011 a las 21:04 -0500, Jonathan Ellis escribió: > "out of sequence response" is thrift's way of saying "I got a response > for request Y when I expected request X." > > my money is on using a single connection from multiple threads. don't do > that. I'm not using thrift direc

Re: RE: batch_mutate failed: out of sequence response

2011-04-06 Thread Jonathan Ellis
"out of sequence response" is thrift's way of saying "I got a response for request Y when I expected request X." my money is on using a single connection from multiple threads. don't do that. On Wed, Apr 6, 2011 at 5:59 PM, Dan Washusen wrote: > Pelops will retry when TimedOutException, TTransp

Re: RE: batch_mutate failed: out of sequence response

2011-04-06 Thread Dan Washusen
Pelops will retry when TimedOutException, TTransportException or UnavailableException exceptions are thrown but not TApplicationException. TApplicationException has a type property which looks like it could be used to retry based on specific values. Based on the names the INTERNAL_ERROR and BAD

Re: RE: batch_mutate failed: out of sequence response

2011-04-06 Thread Héctor Izquierdo Seliva
El mié, 06-04-2011 a las 09:06 +1000, Dan Washusen escribió: > Pelops raises a RuntimeException? Can you provide more info please? > org.scale7.cassandra.pelops.exceptions.ApplicationException: batch_mutate failed: out of sequence response > -- > Dan Washusen > Make big files fly > visit digit

Re: RE: batch_mutate failed: out of sequence response

2011-04-05 Thread Dan Washusen
Pelops raises a RuntimeException? Can you provide more info please? -- Dan Washusen Make big files fly visit digitalpigeon.com On Tuesday, 5 April 2011 at 11:43 PM, Héctor Izquierdo Seliva wrote: El mar, 05-04-2011 a las 09:35 -0400, Dan Hendry escribió: > > I too have seen the out of sequence r

Re: batch_mutate failed: out of sequence response

2011-04-05 Thread Jonathan Ellis
Step 1: disable swap. 2011/4/5 Héctor Izquierdo Seliva : > Update with more info: > > I'm still running into problems. Now I don't write more than 100 columns > at a time, and I'm having lots of Stop-the-world gc pauses. > > I'm writing into three column families, with memtable_operations = 0.3 >

RE: batch_mutate failed: out of sequence response

2011-04-05 Thread Héctor Izquierdo Seliva
Update with more info: I'm still running into problems. Now I don't write more than 100 columns at a time, and I'm having lots of Stop-the-world gc pauses. I'm writing into three column families, with memtable_operations = 0.3 and memtable_throughput = 64. There is now swapping, and full GCs are

RE: batch_mutate failed: out of sequence response

2011-04-05 Thread Héctor Izquierdo Seliva
I'm still running into problems. Now I don't write more than 100 columns at a time, and I'm having lots of Stop-the-world gc pauses. I'm writing into three column families, with memtable_operations = 0.3 and memtable_throughput = 64. Is any of this wrong? > > -Original Message- > > From

RE: batch_mutate failed: out of sequence response

2011-04-05 Thread Héctor Izquierdo Seliva
El mar, 05-04-2011 a las 09:35 -0400, Dan Hendry escribió: > I too have seen the out of sequence response problem. My solution has just > been to retry and it seems to work. None of my mutations are THAT large (< > 200 columns). > > The only related information I could find points to a thrift/u

RE: batch_mutate failed: out of sequence response

2011-04-05 Thread Dan Hendry
I too have seen the out of sequence response problem. My solution has just been to retry and it seems to work. None of my mutations are THAT large (< 200 columns). The only related information I could find points to a thrift/ubuntu bug of some kind (http://markmail.org/message/xc3tskhhvsf5awz7