Re: Permanent ReadTimeout

2015-01-13 Thread Ja Sam
Your response is full of information, after I read it I think that I design something wrong in my system. I will try to present what hardware I have and what I am trying to achieve. *Hardware:* I have 9 machines, every machine has 10 hdd for data (not SSD) and 64 GB of RAM. *Requirements* The Cas

Re: Permanent ReadTimeout

2015-01-13 Thread Eric Stevens
If you have fallen far behind on compaction, this is a hard situation to recover from. It means that you're writing data faster than your cluster can absorb it. The right path forward depends on a lot of factors, but in general you either need more servers or bigger servers, or else you need to w

Re: Permanent ReadTimeout

2015-01-13 Thread Ja Sam
Ad 4) For sure I got a big problem. Because pending tasks: 3094 The question is what should I change/monitor? I can present my whole solution design, if it helps On Mon, Jan 12, 2015 at 8:32 PM, Ja Sam wrote: > To precise your remarks: > > 1) About 30 sec GC. I know that after time my cluster h

Re: Permanent ReadTimeout

2015-01-12 Thread Ja Sam
To precise your remarks: 1) About 30 sec GC. I know that after time my cluster had such problem, we added "magic" flag, but result will be in ~2 weeks (as I presented in screen on StackOverflow). If you have any idea how can fix/diagnose this problem, I will be very grateful. 2) It is probably tr

Re: Permanent ReadTimeout

2015-01-12 Thread Eric Stevens
If you're getting 30 second GC's, this all by itself could and probably does explain the problem. If you're writing exclusively to A, and there are frequent partitions between A and B, then A is potentially working a lot harder than B, because it needs to keep track of hinted handoffs to replay to

Permanent ReadTimeout

2015-01-12 Thread Ja Sam
*Environment* - Cassandra 2.1.0 - 5 nodes in one DC (DC_A), 4 nodes in second DC (DC_B) - 2500 writes per seconds, I write only to DC_A with local_quorum - minimal reads (usually none, sometimes few) *Problem* After a few weeks of running I cannot read any data from my cluster, beca