Point upgrades

2015-09-01 Thread Stan Lemon
I am wondering if when doing a point upgrade, such as 2.0.11 to 2.0.16 do I need to stop all repairs before performing an upgrade on node IF that node is NOT the one running the repair? Basically I would like to upgrade the other nodes, and then when the repair is done I can upgrade that node. Tha

Re: Parallel repairs

2015-08-24 Thread Stan Lemon
Gotcha, we are using vnodes - so I'll go sequentially through both datacenters. Unfortunately that's going to take me two months to complete repairs at this rate. :( Thanks again for your help, SL On Mon, Aug 24, 2015 at 5:17 PM, Robert Coli wrote: > On Mon, Aug 24, 2015 at

Re: Parallel repairs

2015-08-24 Thread Stan Lemon
to go through datacenter 1 and then through datacenter 2, before looping back and starting to walk through the cluster again. Thanks for all your help, Stan On Mon, Aug 17, 2015 at 4:55 PM, Robert Coli wrote: > On Mon, Aug 17, 2015 at 1:37 PM, Stan Lemon wrote: > >> I have not cha

Re: Parallel repairs

2015-08-17 Thread Stan Lemon
On Mon, Aug 17, 2015 at 2:31 PM, Robert Coli wrote: > Have you unthrottled compaction and etc.? 10 days is a long time... > I have not changed the throttle compaction value for our cluster. I've not been sure how to gauge where I can I take this value. Any guidance here would be extremely appre

Re: Parallel repairs

2015-08-17 Thread Stan Lemon
to repair the whole cluster. Thanks, Stan On Fri, Aug 14, 2015 at 2:44 PM, Robert Coli wrote: > On Fri, Aug 14, 2015 at 11:33 AM, Stan Lemon > wrote: > >> Is it safe to run repairs in parallel on multiple nodes in the same DC at >> the time or is this discouraged? >> >

Parallel repairs

2015-08-14 Thread Stan Lemon
Is it safe to run repairs in parallel on multiple nodes in the same DC at the time or is this discouraged? I've got a pretty neglected cluster where repairs have not been run for quite some time and on average I'm seeing them take about 3.5 days to complete per node. Just trying to figure out if I

Re: Long joining node

2015-08-06 Thread Stan Lemon
ays-on, and predictably scalable to any > size. With more than 500 customers in 45 countries, DataStax is the > database technology and transactional backbone of choice for the worlds > most innovative companies such as Netflix, Adobe, Intuit, and eBay. > > On Wed, Aug 5, 2015 at 8:33 AM, S

Re: Long joining node

2015-08-05 Thread Stan Lemon
tabase technology and transactional backbone of choice for the worlds > most innovative companies such as Netflix, Adobe, Intuit, and eBay. > > On Tue, Aug 4, 2015 at 2:21 PM, Stan Lemon wrote: > >> Sebastian, >> You're referring to streaming_socket_timeout_in_ms corre

Re: Long joining node

2015-08-04 Thread Stan Lemon
d’s most innovative enterprises. > Datastax is built to be agile, always-on, and predictably scalable to any > size. With more than 500 customers in 45 countries, DataStax is the > database technology and transactional backbone of choice for the worlds > most innovative companies such as N

Re: Long joining node

2015-08-04 Thread Stan Lemon
:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) On Tue, Aug 4, 2015 at 2:14 PM, Robert Coli wrote: > On Tue, Aug 4, 2015 at 11:02 AM, Stan Lemon wrote

Long joining node

2015-08-04 Thread Stan Lemon
Hello, I have a a cluster with 12 nodes each in 2 datacenters for a total of 24 nodes. I am attempting to add a 13th node in one of the datacenters. I have been monitoring this process from the node itself with nodetool netstats and from one of the existing nodes using nodetool status. On the exi

Re: High cpu usage & segfaulting

2014-11-26 Thread Stan Lemon
You can typically correlate that to increased GC times >> and counts. I'd look at that before looking at strace and such. >> >> Otis >> -- >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >> Solr & Elasticsearch Support * http:/

High cpu usage & segfaulting

2014-11-25 Thread Stan Lemon
We are using v2.0.11 and have seen several instances in our 24 node cluster where the node becomes unresponsive, when we look into it we find that there is a cassandra process chewing up a lot of CPU. There are no other indications in logs or anything as to what might be happening, however if we st

Re: bootstrapping node stuck in JOINING state

2014-11-22 Thread Stan Lemon
Hello, I posted a similar issue the other day. We wound up not nuking the data dir and simply deleting the system keyspace from the data dir and then restarted the node. This actually worked and caused our never-ending join process to complete and the node is now a part of the cluster. Stan

Re: A tale of a node that never joins...

2014-11-19 Thread Stan Lemon
We are currently using 2.0.11 Thanks, Stan > Hello Stan > > Which version of Cassandra are you using ? There are some known issues of > streaming failure that prevent a node from finishing joining > > Regards > > On Wed, Nov 19, 2014 at 3:57 PM, Stan Lemon wrote: >

A tale of a node that never joins...

2014-11-19 Thread Stan Lemon
Hello, I'm working on a two data center cluster with 12 nodes in each data center. I recently wanted to add a thirteenth node to one of the data centers to try and validate some load improvements to our hardware configuration. I added the node following DataStax directions ( http://www.datastax.com