Re: Read query taking a long time

2015-10-21 Thread Carlos Alonso
Well... I think that pretty much is showing the problem. The problem I'd say is a bad data model. Your read query is perfect, it hits a single partition and that's the best situation, but on the other hand, it turns out that there are one or two huge partitions and fully reading them is extremely

Re: Read query taking a long time

2015-10-21 Thread Laing, Michael
Are the clocks synchronized across the cluster - probably, but I thought I would ask :) On Wed, Oct 21, 2015 at 3:35 AM, Brice Figureau < brice+cassan...@daysofwonder.com> wrote: > Hi, > > On 20/10/2015 19:48, Carlos Alonso wrote: > > I think also having the output of cfhistograms could help. I'd

Re: Read query taking a long time

2015-10-21 Thread Brice Figureau
Hi, On 20/10/2015 19:48, Carlos Alonso wrote: > I think also having the output of cfhistograms could help. I'd like to > know how many sstables are being hit during reads. Ooops, my bad I thought you mentioned proxyhistograms. Here's the worst node cfhistograms: nodetool cfhistograms akka messag

Re: Read query taking a long time

2015-10-20 Thread Carlos Alonso
I think also having the output of cfhistograms could help. I'd like to know how many sstables are being hit during reads. Also, which CL are you reading with? cfstats is a local command, so maybe that node you've printed is working fine but there's another that is causing the latency. Can you che

Re: Read query taking a long time

2015-10-20 Thread Brice Figureau
Hi, Thanks for your answer. Unfortunately since I wrote my e-mail, things are a bit better. This might be because I moved from openjdk 7 to oracle jdk 8 after having seen a warning in the C* log about openjdk, and I also added a node (for other reasons). Now the query itself takes only 1.5s~2s i

Re: Read query taking a long time

2015-10-19 Thread Jon Haddad
I wrote a blog post a while back you may find helpful on diagnosing problems in production. There's a lot of potential things that could be wrong with your cluster and going back and forth on the ML to pin down the right one will take forever. http://rustyrazorblade.com/2014/09/cassandra-summi

Re: Read query taking a long time

2015-10-19 Thread Carlos Alonso
Could you send cfhistograms and cfstats relevant to the read column family? That could help Carlos Alonso | Software Engineer | @calonso On 17 October 2015 at 16:15, Brice Figureau < brice+cassan...@daysofwonder.com> wrote: > Hi, > > I've read all I could find on h

Read query taking a long time

2015-10-17 Thread Brice Figureau
Hi, I've read all I could find on how cassandra works, I'm still wondering why the following query takes more than 5s to return on a simple (and modest) 3 nodes cassandra 2.1.9 cluster: SELECT sequence_nr, used FROM messages WHERE persistence_id = 'session-SW' AND partition_nr = 0; The schem