Re: High loads only on one node in the cluster

2013-11-01 Thread Rakesh Rajan
Tyler, Thanks for the explanation. The objective is not to have a perfectly balanced US-East and SG DC clusters. SG DC cluster is just a backup cluster and hence has lesser nodes than US-East cluster. What we are trying to figure out is the imbalance between the 6 nodes within US-East itself. I'll

Re: High loads only on one node in the cluster

2013-11-01 Thread Tyler Hobbs
On Fri, Nov 1, 2013 at 5:07 AM, Rakesh Rajan wrote: > > 1) By alternating racks, do you mean to alternate racks between all nodes > in a single DC v/s multiple DCs? AWS EastCoast has 4 AZs > and Singapore has 2 AZs. So is the final solution something like this: > ip11 - East Coast - m1.xlarge / u

Re: High loads only on one node in the cluster

2013-11-01 Thread Rakesh Rajan
Forgot to mention: All 9 nodes on Cassandra 1.2.9. Also, tpstats on the high CPU node indicate: 1. Pool NameActive Pending Completed Blocked All time blocked 2. ReadStage32 6600 3420385815 0 0 3. RequestRe

Re: High loads only on one node in the cluster

2013-11-01 Thread Rakesh Rajan
@Tyler / @Rob, As Ashish mentioned earlier, we have 9 nodes on AWS - 6 on EastCoast and 3 on Singapore. All 9 nodes uses EC2Snitch. The current ring ( across all nodes in 2 DC ) looks like this: ip11 - East Coast - m1.xlarge / us-east-1b - Size: 83 GB - Token: 0 ip21 - Singapore - m1.xla

Re: High loads only on one node in the cluster

2013-11-01 Thread Ashish Tyagi
Hi Evan, The clients connect to all nodes. We tried shutting the thrift server on the affected node. Loads did not come down. On Fri, Nov 1, 2013 at 12:59 AM, Evan Weaver wrote: > Are all your clients only connecting to your first node? I would > probably strace it and compare the trace to on

Re: High loads only on one node in the cluster

2013-10-31 Thread Robert Coli
On Thu, Oct 31, 2013 at 1:42 PM, Tyler Hobbs wrote: > I think the "Owns" calculation isn't taking racks into consideration. The > fact that you aren't alternating racks (availability zones, with EC2Snitch) > is what is causing the imbalance. I suggest either using the same rack for > all nodes

Re: High loads only on one node in the cluster

2013-10-31 Thread Tyler Hobbs
I think the "Owns" calculation isn't taking racks into consideration. The fact that you aren't alternating racks (availability zones, with EC2Snitch) is what is causing the imbalance. I suggest either using the same rack for all nodes (preferred) or alternate your racks/AZs: 1b, 1c, 1d, 1b, 1c, 1

High loads only on one node in the cluster

2013-10-31 Thread Ashish Tyagi
We have a 9 node cluster. 6 nodes are in one data-center and 3 nodes in the other. All machines are Amazon M1.XLarge configuration. Datacenter: DC1 == Address RackStatus State Load OwnsToken ip11 1b Up Normal 76.46 GB16.67% 0 ip12