Cassandra Scaling Alerts

2014-07-22 Thread Arup Chakrabarti
We have been going through and setting up alerts on our Cassandra clusters. We have catastrophic alerts setup to let us know when things are super broken, but we are now looking at setting up alerts for letting us know when we need to start scaling vertically or horizontally. We have alerts on our

Re: HA Proxy

2014-06-27 Thread Arup Chakrabarti
We tried using HAProxy (to force connections to the closest datacenter) instead of relying on Hector to do the right thing and had problems with it. The client libraries we have used are smart enough to handle the node up/down case, and in most scenarios, if there is a slow node, they will route ar

Re: High latency on 5 node Cassandra Cluster

2014-06-05 Thread Arup Chakrabarti
# cass12: us-west-2b private.ip=DC-B:2 public.ip=DC-B:2 We're figuring out the upgrade right now. In general, what is the expected upgrade time per node for ~50GB (on each node)? Thanks arup On Wed, Jun 4, 2014 at 11:08 AM, Robert Coli wrote: > On Wed, Jun 4, 2014 at 12:12

High latency on 5 node Cassandra Cluster

2014-06-04 Thread Arup Chakrabarti
Hello. We had some major latency problems yesterday with our 5 node cassandra cluster. Wanted to get some feedback on where we could start to look to figure out what was causing the issue. If there is more info I should provide, please let me know. Here are the basics of the cluster: Clients: Hect