Re: Cassandra read throughput with little/no caching.

2013-01-02 Thread James Masson
On 02/01/13 16:18, Tyler Hobbs wrote: On Wed, Jan 2, 2013 at 5:28 AM, James Masson mailto:james.mas...@opigram.com>> wrote: > 1) Hector sends a request to some node in the cluster, which will act as the coordinator. 2) The coordinator then sends the actual read requests out to ea

Re: Cassandra read throughput with little/no caching.

2013-01-02 Thread James Masson
On 31/12/12 18:45, Tyler Hobbs wrote: On Mon, Dec 31, 2012 at 11:24 AM, James Masson mailto:james.mas...@opigram.com>> wrote: Well, it turns out the Read-Request Latency graph in Ops-Center is highly misleading. Using jconsole, the read-latency for the column family in qu

Re: Cassandra read throughput with little/no caching.

2012-12-31 Thread James Masson
left on my list for investigation are memtable sizes / eviction and JNA - and trying to capture some of the requests that are causing the spikes for further investigation. James M On 31/12/12 10:05, James Masson wrote: Hi Yiming, I've had the chance to observe what happens to cassandr

Re: Cassandra read throughput with little/no caching.

2012-12-31 Thread James Masson
Hi Yiming, I've had the chance to observe what happens to cassandra read response time over time. It starts out with fast 1ms reads, until the first compaction starts, then the CPUs are maxed out for a period, and read latency rises to 4ms. After compaction finishes, the system returns to 1

Re: Cassandra read throughput with little/no caching.

2012-12-24 Thread James Masson
On 21/12/12 17:56, Yiming Sun wrote: James, you could experiment with Row cache, with off-heap JNA cache, and see if it helps. My own experience with row cache was not good, and the OS cache seemed to be most useful, but in my case, our data space was big, over 10TB. Your sequential access pa

Re: Cassandra read throughput with little/no caching.

2012-12-24 Thread James Masson
Hi Aaron, On 23/12/12 20:18, aaron morton wrote: First, the non helpful advice, I strongly suggest changing the data model so you do not have 100MB+ rows. They will make life harder. I don't think we have 100MB+ rows. Column families, yes - but not rows. Write request latency is about 900

Re: Cassandra read throughput with little/no caching.

2012-12-21 Thread James Masson
(http://lmax-exchange.github.com/disruptor/) fed to do the processing work. But this all doesn't change the fact that under this zero-cache workload, cassandra seems to be very CPU expensive for throughput. thanks James M On Fri, Dec 21, 2012 at 10:36 AM, James Masson mailto:james.mas...@

Re: Cassandra read throughput with little/no caching.

2012-12-21 Thread James Masson
nux disk cache. 2Gb VM, 512Mb cassandra heap - GCs are nice and quick, no JVM memory problems, used heap oscillates between 280-350Mb. Basically, I'm just puzzled as cassandra doesn't behave as I would expect. Huge CPU use in cassandra for very little throughput. I'm struggling t

Cassandra read throughput with little/no caching.

2012-12-21 Thread James Masson
Hi list-users, We have an application that has a relatively unusual access pattern in cassandra 1.1.6 Essentially we read an entire multi hundred megabyte column family sequentially (little chance of a cassandra cache hit), perform some operations on the data, and write the data back to ano