Re: Nodes Dying in 2.1.2

2015-01-03 Thread Phil Burress
t; > =Rob > > On Mon, Dec 29, 2014 at 1:40 PM, Robert Coli wrote: > >> On Wed, Dec 24, 2014 at 9:41 AM, Phil Burress >> wrote: >> >>> Just upgraded our cluster from 2.1.1 to 2.1.2 and our nodes keep dying. >>> The kernel is

Re: 答复: Downgrade from 2.1.2 to 2.1.1

2014-12-31 Thread Phil Burress
-- > > 李建奇 > > 京东商城 【 运营研发 架构师】 > > = > > 手机:18600360358 > > 地址:北京市大兴区 朝林广场25层 > > 邮编:100101 > > = > > 网购上京东,省钱又放心! > > www.jd.com > > > > *发件人:* Phil Burress [mailto:ph

Re: Downgrade from 2.1.2 to 2.1.1

2014-12-30 Thread Phil Burress
Thanks Rob. On Tue, Dec 30, 2014 at 1:38 PM, Robert Coli wrote: > On Tue, Dec 30, 2014 at 9:42 AM, Phil Burress > wrote: > >> We are having a lot of problems with release 2.1.2. It was suggested here >> we should downgrade to 2.1.1 if possible. >> >> For the

Downgrade from 2.1.2 to 2.1.1

2014-12-30 Thread Phil Burress
We are having a lot of problems with release 2.1.2. It was suggested here we should downgrade to 2.1.1 if possible. For the experts out there, do you foresee any issues in doing this? Thanks! Phil

Nodes Dying in 2.1.2

2014-12-24 Thread Phil Burress
Just upgraded our cluster from 2.1.1 to 2.1.2 and our nodes keep dying. The kernel is killing the process due to out of memory: kernel: Out of memory: Kill process 6267 (java) score 998 or sacrifice child Appears to only occur during compactions. We've tried playing with the heap settings but no

Re: nodetool repair -snapshot option?

2014-07-01 Thread Phil Burress
//github.com/hancockks/cassandra_range_repair > > > > > On Mon, Jun 30, 2014 at 10:20 PM, Phil Burress > wrote: > >> @Paulo, this is very cool! Thanks very much for the link! >> >> >> On Mon, Jun 30, 2014 at 9:37 PM, Paulo Ricardo Motta Gomes < >> paulo

Re: nodetool repair -snapshot option?

2014-06-30 Thread Phil Burress
el. :) > > > On Mon, Jun 30, 2014 at 10:26 PM, Phil Burress > wrote: > >> One last question. Any tips on scripting a subrange repair? >> >> >> On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress >> wrote: >> >>> We are running repair -pr. We

Re: nodetool repair -snapshot option?

2014-06-30 Thread Phil Burress
One last question. Any tips on scripting a subrange repair? On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress wrote: > We are running repair -pr. We've tried subrange manually and that seems to > work ok. I guess we'll go with that going forward. Thanks for all the info! > >

Re: nodetool repair -snapshot option?

2014-06-30 Thread Phil Burress
We are running repair -pr. We've tried subrange manually and that seems to work ok. I guess we'll go with that going forward. Thanks for all the info! On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia < chovatia.jayd...@gmail.com> wrote: > Are you running full repair or on subset? If you are run

nodetool repair -snapshot option?

2014-06-30 Thread Phil Burress
We are running into an issue with nodetool repair. One or more of our nodes will die with OOM errors when running nodetool repair on a single node. Was reading this http://www.datastax.com/dev/blog/advanced-repair-techniques and it mentioned using the -snapshot option, however, that doesn't appear

Ec2 Network I/O

2014-05-19 Thread Phil Burress
Has anyone experienced network i/o issues with ec2? We are seeing a lot of these in our logs: HintedHandOffManager.java (line 477) Timed out replaying hints to /10.0.x.xxx; aborting (15 delivered) and these... Cannot handshake version with /10.0.x.xxx and these... java.io.IOException: Cannot p

Re: Recommended Approach for Config Changes

2014-04-28 Thread Phil Burress
pr 25, 2014, at 8:59 AM, Tyler Hobbs wrote: > > > On Fri, Apr 25, 2014 at 10:43 AM, Phil Burress > wrote: > >> Thanks. I made a change to a single node and it took almost an hour to >> rejoin the cluster (go from DN to UP in nodetool status). The cluster is >

Re: Recommended Approach for Config Changes

2014-04-25 Thread Phil Burress
e changes you can manually have take affect without a restart (ie > compactionthroughput, things settable from jmx). There is also config > changes you cant really make like switching the snitch and such without a > big todo. > > --- > Chris > > On Apr 25, 2014, at 8:53 AM,

Re: Bootstrap Timing

2014-04-25 Thread Phil Burress
Cassandra 2.0.6 On Fri, Apr 25, 2014 at 10:31 AM, James Rothering wrote: > What version of C* is this? > > > On Fri, Apr 25, 2014 at 6:55 AM, Phil Burress wrote: > >> Just a follow-up on this for any interested parties. Ultimately we've >> determined that the boo

Re: Bootstrap Timing

2014-04-25 Thread Phil Burress
Just a follow-up on this for any interested parties. Ultimately we've determined that the bootstrap/join process is broken in Cassandra. We ended up creating an entirely new cluster and migrating the data. On Mon, Apr 21, 2014 at 10:32 AM, Phil Burress wrote: > The new node has managed

Recommended Approach for Config Changes

2014-04-25 Thread Phil Burress
If I wanted to make a configuration change to a single node in a cluster, what is the recommended approach for doing that? Is it ok to just stop that instance, make the change and then restart it?

Re: Bootstrap Timing

2014-04-21 Thread Phil Burress
it should take, but I'd think that others on the list with >> similar database sizes might be able to give you a better idea. >> >> Steve >> >> >> >> On Fri, Apr 18, 2014 at 1:43 PM, Phil Burress >> wrote: >> >>> First, I just stop

Re: Bootstrap Timing

2014-04-19 Thread Phil Burress
lar database sizes might be able to give you a better idea. > > Steve > > > > On Fri, Apr 18, 2014 at 1:43 PM, Phil Burress wrote: > >> First, I just stopped 2 of the nodes and left one running. But this >> morning, I stopped that third node, cleared out the data, resta

Re: Bootstrap Timing

2014-04-18 Thread Phil Burress
to continue? > > Steve > > > On Fri, Apr 18, 2014 at 10:40 AM, Robert Coli wrote: > >> On Fri, Apr 18, 2014 at 5:05 AM, Phil Burress >> wrote: >> >>> nodetool netstats shows 84 files. They are all at 100%. Nothing showing >>> in Pending or Active

Re: Bootstrap Timing

2014-04-18 Thread Phil Burress
On Thu, Apr 17, 2014 at 4:32 PM, Robert Coli wrote: > On Wed, Apr 16, 2014 at 1:56 PM, Phil Burress wrote: > >> I've shut down two of the nodes and am bootstrapping one right now. Is >> there any way to tell when it will finish bootstrapping? >> > > "nodetoo

Re: Bootstrap Timing

2014-04-16 Thread Phil Burress
I've shut down two of the nodes and am bootstrapping one right now. Is there any way to tell when it will finish bootstrapping? On Wed, Apr 16, 2014 at 2:56 PM, Phil Burress wrote: > Thanks! > > > On Wed, Apr 16, 2014 at 2:50 PM, Robert Coli wrote: > >> On Wed, Apr

Re: Bootstrap Timing

2014-04-16 Thread Phil Burress
Thanks! On Wed, Apr 16, 2014 at 2:50 PM, Robert Coli wrote: > On Wed, Apr 16, 2014 at 11:16 AM, Phil Burress > wrote: > >> Thanks very much for the response. I'm not using vnodes, does that >> matter? >> > > Not in your case. In some cases it is saf

Re: Bootstrap Timing

2014-04-16 Thread Phil Burress
Also, one more quick question. For the new nodes, do I add all three existing nodes as seeds? Or just add one? On Wed, Apr 16, 2014 at 2:16 PM, Phil Burress wrote: > Thanks very much for the response. I'm not using vnodes, does that matter? > > > On Wed, Apr 16, 2014 at 2:1

Re: Bootstrap Timing

2014-04-16 Thread Phil Burress
Thanks very much for the response. I'm not using vnodes, does that matter? On Wed, Apr 16, 2014 at 2:13 PM, Robert Coli wrote: > On Wed, Apr 16, 2014 at 11:10 AM, Phil Burress > wrote: > >> How long does bootstrapping typically take? I have 3 existing nodes in >> o

Bootstrap Timing

2014-04-16 Thread Phil Burress
Greetings, How long does bootstrapping typically take? I have 3 existing nodes in our cluster with about 40GB each. I've added three new nodes to the cluster. They have been in bootstrap mode for a little over 3 days now. Should I be concerned? Is there a way to tell how long it will take to finis