Re: High disk I/O during reads

2013-03-25 Thread Jon Scarborough
y is over 100 ms which is really > really slow. I am seeing less than 3ms reads for my cluster which is on > SSD. > Can you also check the nodetool cfhistorgram, it tells you more about the > number of SSTable involved and read/write latency. Somtimes average doesn't > tell you t

Re: High disk I/O during reads

2013-03-24 Thread aaron morton
D. >>>> Can you also check the nodetool cfhistorgram, it tells you more about the >>>> number of SSTable involved and read/write latency. Somtimes average doesn't >>>> tell you the whole storey. >>>> Also check your nodetool tpstats, are there a l

Re: High disk I/O during reads

2013-03-23 Thread Matt Kap
eads for my cluster which is on SSD. >>> Can you also check the nodetool cfhistorgram, it tells you more about the >>> number of SSTable involved and read/write latency. Somtimes average doesn't >>> tell you the whole storey. >>> Also check your nodetool

Re: High disk I/O during reads

2013-03-23 Thread igor
average >> doesn't tell you the whole storey. >> Also check your nodetool tpstats, are there a lot dropped reads? >> >> -Wei >> - Original Message - >> From: "Jon Scarborough" >> To: user@cassandra.apache.org >> Sent: Friday, M

RE: High disk I/O during reads

2013-03-22 Thread Kanwar Sangha
: High disk I/O during reads Checked tpstats, there are very few dropped messages. Checked histograms. Mostly nothing surprising. The vast majority of rows are small, and most reads only access one or two SSTables. What I did discover is that of our 5 nodes, one is performing well, with disk I/O

Re: High disk I/O during reads

2013-03-22 Thread Jon Scarborough
h" > To: user@cassandra.apache.org > Sent: Friday, March 22, 2013 9:42:34 AM > Subject: Re: High disk I/O during reads > > Key distribution across probably varies a lot from row to row in our case. > Most reads would probably only need to look at a few SSTables, a few mig

Re: High disk I/O during reads

2013-03-22 Thread Wei Zhu
doesn't tell you the whole storey. Also check your nodetool tpstats, are there a lot dropped reads? -Wei - Original Message - From: "Jon Scarborough" To: user@cassandra.apache.org Sent: Friday, March 22, 2013 9:42:34 AM Subject: Re: High disk I/O during reads Key distribution

Re: High disk I/O during reads

2013-03-22 Thread Jon Scarborough
ache.org<mailto:user@cassandra.apache.org>> > Date: Friday, March 22, 2013 8:38 AM > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Subject: RE: High disk I/O during reads &

RE: High disk I/O during reads

2013-03-22 Thread Kanwar Sangha
apache.org Subject: Re: High disk I/O during reads Did you mean to ask "are 'all' your keys spread across all SSTables"? I am guessing at your intention. I mean I would very well hope my keys are spread across all sstables or otherwise that sstable should not be there as he has

Re: High disk I/O during reads

2013-03-22 Thread Hiller, Dean
38 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: RE: High disk I/O during reads Are your Keys spread across all SSTables ? That will cause every sstable read which will increase the I/O. What compaction are

RE: High disk I/O during reads

2013-03-22 Thread Kanwar Sangha
: High disk I/O during reads Hello, We've had a 5-node C* cluster (version 1.1.0) running for several months. Up until now we've mostly been writing data, but now we're starting to service more read traffic. We're seeing far more disk I/O to service these reads than I would ha

High disk I/O during reads

2013-03-21 Thread Jon Scarborough
Hello, We've had a 5-node C* cluster (version 1.1.0) running for several months. Up until now we've mostly been writing data, but now we're starting to service more read traffic. We're seeing far more disk I/O to service these reads than I would have anticipated. The CF being queried consists of