Re: ReplicateOnWrite issues

2011-07-12 Thread Sylvain Lebresne
On Wed, Jul 13, 2011 at 1:00 AM, David Hawthorne wrote: > Thanks for looking at that. > > Our use case involves supercolumns that have 2-20,000 counters within them.   > For a set of continuous updates to one supercolumn, the behavior you're > describing is: Here's your problem. Don't do that. I

Re: ReplicateOnWrite issues

2011-07-12 Thread Sylvain Lebresne
On Wed, Jul 13, 2011 at 12:18 AM, David Hawthorne wrote: > > On Jul 12, 2011, at 3:02 PM, Sylvain Lebresne wrote: > >> On Tue, Jul 12, 2011 at 11:42 PM, David Hawthorne >> wrote: >>> Well, I was using a large number of clients:  I tried configuring a hector >>> pool of 20-200 to see what affect

Re: ReplicateOnWrite issues

2011-07-12 Thread David Hawthorne
On Jul 12, 2011, at 3:02 PM, Sylvain Lebresne wrote: > On Tue, Jul 12, 2011 at 11:42 PM, David Hawthorne > wrote: >> Well, I was using a large number of clients: I tried configuring a hector >> pool of 20-200 to see what affect that had on throughput. There's definitely >> a point after whic

Re: ReplicateOnWrite issues

2011-07-12 Thread Sylvain Lebresne
On Tue, Jul 12, 2011 at 11:42 PM, David Hawthorne wrote: > Well, I was using a large number of clients:  I tried configuring a hector > pool of 20-200 to see what affect that had on throughput. There's definitely > a point after which there's no gain, so I dialed it back down.  To clarify a > f

Re: ReplicateOnWrite issues

2011-07-12 Thread David Hawthorne
Well, I was using a large number of clients: I tried configuring a hector pool of 20-200 to see what affect that had on throughput. There's definitely a point after which there's no gain, so I dialed it back down. To clarify a few other things, when I say inserts I mean increments, as this te

Re: ReplicateOnWrite issues

2011-07-12 Thread Sylvain Lebresne
When you do counter increment at CL.ONE, the write is acknowledged as soon as the first replica getting the the write has pushed the increment into his memtable. However, there is a read happening for the replication to the other replicas (this is necessary to the counter design). What is happening

Re: ReplicateOnWrite issues

2011-07-12 Thread Yang
what do you mean by "until they are full" ? right now I guess a quick black-box testing method for this problem is to try inserting only shorter rows , and see if that persists. as you said, it could be that addReadCommandFromColumnFamily is taking a lot of time to read, if that's from disk, it's

Re: ReplicateOnWrite issues

2011-07-12 Thread David Hawthorne
It's definitely for counters, and some of the rows I'm inserting are long-ish, if 1.3MB is long. Maybe it would help if I said I was using counter super columns. I'm also writing to only a handful of rows at a time, until they are full. It looks like the counter super column code in addReadCo

Re: ReplicateOnWrite issues

2011-07-12 Thread Yang
interesting, first just to make sure: since replicateOnWrite is for Counters, you are using counters (you use the word "insert" instead of "add/increment" ) right? if you are using counters, supposedly the leader runs replicateOnWrite, somehow all your adds find the one box as leader, that's prob

ReplicateOnWrite issues

2011-07-11 Thread David Hawthorne
This is now my fourth attempt to get the message through. Apologies if you see multiple copies. I've tried to give as much relevant data as I can think of, but please let me know if you need any other info. I spent the day getting jmxtrans to talk to statsd with the cassandra JMX data, so I c