Re: Repairing lost data

2011-08-27 Thread Anthony Molinaro
I'm pretty sure that was a bug fixed in a later 0.6.x release so you might be able to upgrade and the exceptions might go away. We run 0.6.13 with a minor mod to support data expiration and will probably do so indefinitely since there no way to upgrade without shutting our site down :( -Anthon

Re: Upgrading from 0.6 to 0.7.0

2011-01-21 Thread Anthony Molinaro
Port and inter-server port (is this the storage_port?) > >>- Start them up > >>- Upgrade clients one by one > >>- Upgrade the the rest of the servers > >> > >> Or might we get some kind of data collision when still writing to the old > >> cluster as the new storage is being used? > >> > >> /Daniel > >> > >> > > -- Anthony Molinaro

Re: Upgrading from 0.6 to 0.7.0

2011-01-19 Thread Anthony Molinaro
f both the clients and the server. -Anthony On Wed, Jan 19, 2011 at 05:05:02PM -0800, Anthony Molinaro wrote: > Really, my bad, I though they were, but maybe I'm confusing that with > protobuf, I work with too many serialization formats :(. > > -Anthony > > On Wed, Jan 19, 2

Re: Upgrading from 0.6 to 0.7.0

2011-01-19 Thread Anthony Molinaro
Really, my bad, I though they were, but maybe I'm confusing that with protobuf, I work with too many serialization formats :(. -Anthony On Wed, Jan 19, 2011 at 04:46:48PM -0600, Jonathan Ellis wrote: > On Wed, Jan 19, 2011 at 4:34 PM, Anthony Molinaro > wrote: > > Actually I d

Re: Upgrading from 0.6 to 0.7.0

2011-01-19 Thread Anthony Molinaro
n Wed, Jan 19, 2011 at 10:04:54PM +, Aaron Morton wrote: > Your right, forgot about the change to binary keys :) > > Forgot what I said.  > A > > > On 20 Jan, 2011,at 11:01 AM, Anthony Molinaro > wrote: > > As far as I can tell, it is impossible to run

Re: Upgrading from 0.6 to 0.7.0

2011-01-19 Thread Anthony Molinaro
ow did you go about? > > Is there at least a way to avoid having to upgrade both server side and > client side simultaneously? > > Thanks, > Daniel > -- Anthony Molinaro

Re: Cold boot performance problems

2010-10-08 Thread Anthony Molinaro
On Fri, Oct 08, 2010 at 05:31:28PM -0700, Dave Viner wrote: > Has anyone found solid step-by-step docs on how to raid0 the ephemeral disks > in ec2 for use by Cassandra? No, but here's a script I used to raid0 3 ephemerals in an xlarge instance. You can edit the top part for different configs for

Re: Data Center Move

2010-09-02 Thread Anthony Molinaro
cluster was up and running on the > public addresses, you should be able to use those public addresses > to migrate to some other site outside of EC2. > > Am I missing something obvious? (Quite possible, since I haven't actually > tested this) > > On Thu, Sep 02

Data Center Move

2010-09-02 Thread Anthony Molinaro
appen? Thanks, -Anthony -- ---- Anthony Molinaro

Re: Cassandra & HAProxy

2010-08-29 Thread Anthony Molinaro
On Sun, Aug 29, 2010 at 12:20:10PM -0700, Benjamin Black wrote: > On Sun, Aug 29, 2010 at 11:04 AM, Anthony Molinaro > wrote: > > > > > > I don't know it seems to tax our setup of 39 extra large ec2 nodes, its > > also closer to 24000 reqs/sec at peak sinc

Re: Cassandra & HAProxy

2010-08-29 Thread Anthony Molinaro
On Sat, Aug 28, 2010 at 02:44:41PM -0700, Benjamin Black wrote: > On Sat, Aug 28, 2010 at 2:34 PM, Anthony Molinaro > wrote: > > I think maybe he thought you meant put a layer between cassandra internal > > communication. > > No, I took the question to be about client co

Re: Cassandra & HAProxy

2010-08-28 Thread Anthony Molinaro
gt;> > any reason on why loadbalancing client connections using HAProxy isnt > recommended? -- Anthony Molinaro

Re: what causes MESSAGE-DESERIALIZER-POOL to spike

2010-07-26 Thread Anthony Molinaro
:+UseConcMark > 24933 root 40 0 20.6G 8345M 6883M S 0.0 69.4 0:22.57 > /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark > 24939 root 40 0 20.6G 8345M 6883M S 0.0 69.4 0:12.73 > /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark > 25280 root 40 0 20.6G 8345M 6883M S 0.0 69.4 0:00.10 > /opt/java/bin/java -ea -Xms1G -Xmx7G -XX:+UseParNewGC -XX:+UseConcMark -- Anthony Molinaro

Re: Bootstrap question

2010-07-21 Thread Anthony Molinaro
ng IP was previously visible on the cluster), but it seems > very plausible. > > Any chance you could apply the patch for 1221 and test? > > Gary. > > On Tue, Jul 20, 2010 at 16:45, Anthony Molinaro > wrote: > > I see this in the old nodes > > > > DEBUG [WRI

Re: Bootstrap question

2010-07-20 Thread Anthony Molinaro
after nuking those. Better > snapshot the system keyspace first, just in case. > > On Sun, Jul 18, 2010 at 2:01 PM, Anthony Molinaro > wrote: > > Yeah, I tried all that already and it didn't seem to work, no new nodes > > will bootstrap, which makes me think there

Re: Bootstrap question

2010-07-18 Thread Anthony Molinaro
new > > nodes and do a repair, would that actually work?). > > > > Anyone seen this sort of issue? All this is with 0.6.3 so I assume > > eventually others will see this issue. > > > > -Anthony > > > > On Thu, Jul 15, 2010 at 10:45:08PM -0700, Antho

Re: Bootstrap question

2010-07-17 Thread Anthony Molinaro
n Thu, Jul 15, 2010 at 10:45:08PM -0700, Anthony Molinaro wrote: > Okay, so things were pretty messed up. I shut down all the new nodes, > then the old nodes started doing the half the ring is down garbage which > pretty much requires a full restart of everything. So I had to shut >

Bad SSTable killing compaction?

2010-07-17 Thread Anthony Molinaro
anks, -Anthony -- -------- Anthony Molinaro

Re: Bootstrap question

2010-07-16 Thread Anthony Molinaro
On Thu, Jul 15, 2010 at 10:45:08PM -0700, Anthony Molinaro wrote: > Is there something else I should try? The only thing I can think of > is deleting the system directory on the new node, and restarting, so > I'll try that and see if it does anything. So I tried this, it didn

Re: Bootstrap question

2010-07-15 Thread Anthony Molinaro
g. -Anthony On Thu, Jul 15, 2010 at 03:43:49PM -0500, Jonathan Ellis wrote: > On Thu, Jul 15, 2010 at 3:28 PM, Anthony Molinaro > wrote: > > Is the fact that 2 new nodes are in the range messing it up? > > Probably. > > >  And if so > > how do I recover (I'm

Re: Bootstrap question

2010-07-15 Thread Anthony Molinaro
Oh, and looking at the load on the new machines it appears that New 2 and New 6 have gotten some data (although neither is in the ring yet). Not sure if that clears anything up though. -Anthony On Thu, Jul 15, 2010 at 01:28:06PM -0700, Anthony Molinaro wrote: > This is a cluster which

Re: Bootstrap question

2010-07-15 Thread Anthony Molinaro
what token it is going to bootstrap to. Who owns the > ranges that contain those tokens? > > On Wed, Jul 14, 2010 at 5:58 PM, Anthony Molinaro > wrote: > > Hi, > > > >  I have a 0.6.3 cluster which contains 6 nodes.  I added 6 new nodes > > by setting Au

Bootstrap question

2010-07-14 Thread Anthony Molinaro
r things I should be looking at? Thanks, -Anthony -- -------- Anthony Molinaro

Re: Cassandra on AWS across Regions

2010-06-29 Thread Anthony Molinaro
IP address > instead their Private IP. This is not a problem if we setup the entire > cluster with in the same region and zones (US-EAST), cuz they can communicate > with private IP. Can any one share What is the recommended approach here? > > > > Lenin > > > -- > twitter: leningali > skype: galilenin > Cell:513.382.3371 > > > > > > -- Anthony Molinaro

JVM Options for Production

2010-06-14 Thread Anthony Molinaro
oes it very greatly from installation to installation? Thanks, -Anthony -- -------- Anthony Molinaro

Re: GMFD messages

2010-05-27 Thread Anthony Molinaro
ge made me think it might (as only the nodes with a backed up MessageDeserializer had that message). Do gossip messages flow through the MessageDeserializer? Thanks for the response, -Anthony > On Tue, May 25, 2010 at 5:33 PM, Anthony Molinaro > wrote: > > Hi, > > > >  I

GMFD messages

2010-05-25 Thread Anthony Molinaro
be causing this sort of thing? This cluster is now at 27 m1.xlarge boxes on ec2 running 0.6.2 of some flavor. Thanks, -Anthony -- Anthony Molinaro

Re: Compaction JMX Stats?

2010-05-21 Thread Anthony Molinaro
) there is nothing special about > compactions wrt deserialize. Is there some metric I could look at to determine if it's a? CPU is usually at most 20% on these boxes, so I assume it's a causing the backup. -Anthony > On Thu, May 20, 2010 at 12:33 PM, Anthony Molinaro > wrote: &

Compaction JMX Stats?

2010-05-20 Thread Anthony Molinaro
-Anthony -- -------- Anthony Molinaro

Re: Overfull node

2010-05-17 Thread Anthony Molinaro
;> > >>> > >>>>I've got two (out of five) nodes on my cassandra ring that somehow got > >>>>too > >>>>full (e.g. over 60% disk space utilization). I've now gotten a few new > >>>>machines added to the ring, but evertime one of the overfull nodes > >>>>attempts > >>>>to stream its data it runs out of diskspace... I've tried half a dozen > >>>>different bad ideas of how to get things moving along a bit smoother, > >>>>but > >>>>am > >>>>at a total loss at this point. > >>>> > >>>>Is there any good tricks to get cassandra to not need 2x the disk space > >>>>to > >>>>stream out, or is something else potentially going on that's causing me > >>>>problems? > >>>> > >>>>Thanks, > >>>> > >>>> > >>>> > >>> > >>> > >>> > >> > > > > > > -- Anthony Molinaro

nodetool drain disables writes?

2010-05-11 Thread Anthony Molinaro
ere frozen, ls of commit log directory showed no files), is that expected behavior? Is that command documented anywhere on the wiki? Thanks, -Anthony -- -------- Anthony Molinaro

Re: Error in TBaseHelper compareTo(byte [] a , byte [] b)

2010-05-03 Thread Anthony Molinaro
#x27;t noticed any oddities. -Anthony -- -------- Anthony Molinaro

Re: Batch mutate doesn't work

2010-04-30 Thread Anthony Molinaro
t;>>, [ #mutation { column_or_supercolumn = #columnOrSuperColumn { column = #column { name = "property", value = "value", timestamp = 2 } } } ] } }, Try that, -Anthony -- Anthony Molinaro

Re: Can Cassandra make real use of several DataFileDirectories?

2010-04-26 Thread Anthony Molinaro
> > > RAID0 is designed specifically to improve performance (both latency > > and bandwidth). I'm unclear about why you think it would decrease > > importance. Perhaps you're thinking of another RAID type? > > > > Paul Prescod > > -- Anthony Molinaro

Cassandra Job in Pasadena

2010-04-26 Thread Anthony Molinaro
ave 3 cassandra clusters. Feel free to email me offlist any questions you might have, and if you are interested please send your resume. Thanks, -Anthony -- -------- Anthony Molinaro

Re: Odd ring problems with 0.5.1

2010-04-23 Thread Anthony Molinaro
0 at 02:22:11PM -0700, Anthony Molinaro wrote: > > On Fri, Apr 23, 2010 at 01:17:21PM -0500, Jonathan Ellis wrote: > > On Fri, Apr 23, 2010 at 1:12 PM, Anthony Molinaro > > wrote: > > > I'm not sure how it would get this, maybe I need to restart my seed node? > &g

Re: Odd ring problems with 0.5.1

2010-04-23 Thread Anthony Molinaro
On Fri, Apr 23, 2010 at 01:17:21PM -0500, Jonathan Ellis wrote: > On Fri, Apr 23, 2010 at 1:12 PM, Anthony Molinaro > wrote: > > I'm not sure how it would get this, maybe I need to restart my seed node? > > It's worth a try. Sounds like you found an unusual bug in g

Re: Odd ring problems with 0.5.1

2010-04-23 Thread Anthony Molinaro
On Fri, Apr 23, 2010 at 12:41:17PM -0500, Jonathan Ellis wrote: > On Fri, Apr 23, 2010 at 12:30 PM, Anthony Molinaro > wrote: > > Some nodes appear in the ring from some nodes, but not others.  Right > > now I have 14 nodes, 10 of those nodes have the same output of a >

Odd ring problems with 0.5.1

2010-04-23 Thread Anthony Molinaro
and it has not gone down. The seed node has all nodes. Anyone seen this? How can I get those 4 nodes to see the missing node? If a known issue has it been fixed in 0.6 or newer? Thanks, -Anthony -- -------- Anthony Molinaro

Re: Cassandra 0.5.1 restarts slow

2010-04-21 Thread Anthony Molinaro
On Wed, Apr 21, 2010 at 01:24:45PM -0500, Jonathan Ellis wrote: > On Wed, Apr 21, 2010 at 1:11 PM, Anthony Molinaro > wrote: > > Interesting, in the config I see > > > >   > >  5000 > > > > So I thought that timeout was for inter-node communication

Re: Clarification on Ring operations in Cassandra 0.5.1

2010-04-21 Thread Anthony Molinaro
On Wed, Apr 21, 2010 at 12:05:07PM -0500, Jonathan Ellis wrote: > On Wed, Apr 21, 2010 at 11:31 AM, Anthony Molinaro > wrote: > > > > On Wed, Apr 21, 2010 at 11:08:19AM -0500, Jonathan Ellis wrote: > >> Yes, that looks right, where "token really close" means

Re: Cassandra 0.5.1 restarts slow

2010-04-21 Thread Anthony Molinaro
On Wed, Apr 21, 2010 at 12:52:32PM -0500, Jonathan Ellis wrote: > On Wed, Apr 21, 2010 at 12:45 PM, Anthony Molinaro > wrote: > >> as for why it backs up in the first place before the restart, you can > >> either (a) throttle writes [set your timeout lower, make yo

Re: Cassandra 0.5.1 restarts slow

2010-04-21 Thread Anthony Molinaro
adding capacity to match demand. > > On Tue, Apr 20, 2010 at 4:57 PM, Anthony Molinaro > wrote: > > Hi, > > > >  I have a cassandra cluster where a couple things are happening.  Every > > once in a while a node will start to get backed up.  Checking tpstats I > &

Re: Clarification on Ring operations in Cassandra 0.5.1

2010-04-21 Thread Anthony Molinaro
ly wondering if I could speed things up by scping the sstables while the system was running (since they shouldn't be changing). Then in quick succession removetoken and bootstrap with the old token. Probably grasping at straws here :b Thanks for the answers, -Anthony > On Wed, Apr 21, 2

Re: Clarification on Ring operations in Cassandra 0.5.1

2010-04-21 Thread Anthony Molinaro
-0700, Anthony Molinaro wrote: > > On Mon, Apr 19, 2010 at 03:28:26PM -0500, Jonathan Ellis wrote: > > > Can I then 'nodeprobe move ', and > > > achieve the same as step 2 above? > > > > You can't have two nodes with the same token in the ring

Re: Clarification on Ring operations in Cassandra 0.5.1

2010-04-19 Thread Anthony Molinaro
ight? Once I've run bootstrap with the InitialToken value set in the config is it then ignored in subsequent restarts, and if so can I just remove it after that first time? Thanks, -Anthony -- Anthony Molinaro

Clarification on Ring operations in Cassandra 0.5.1

2010-04-15 Thread Anthony Molinaro
stion, if I bring up a new node with AutoBootstrap = false, what happens? Does it join the ring but without data and without token range? Can I then 'nodeprobe move ', and achieve the same as step 2 above? Thanks, -Anthony -- -------- Anthony Molinaro

Re: Recovery from botched compaction

2010-04-13 Thread Anthony Molinaro
On Tue, Apr 13, 2010 at 10:54:51AM -0500, Jonathan Ellis wrote: > On Sat, Apr 10, 2010 at 2:24 PM, Anthony Molinaro > wrote: > >  This is sort of a pre-emptive question as the compaction I'm doing hasn't > > failed yet but I expect it to any time now.  I have a cluste

Recovery from botched compaction

2010-04-10 Thread Anthony Molinaro
space be able to anti-compact without filling up? or does anti-compaction like compaction require 2x disk space? Thanks, -Anthony -- -------- Anthony Molinaro