Re: How to determine if repair need to be run

2011-03-31 Thread Eric Gilmore
Peter, I want to join everyone else thanking you for helping out so much with this thread, and especially for pointing out the problems with the DS docs on this topic. We have some corrections posted today, and will keep looking to improve the information. On Thu, Mar 31, 2011 at 3:11 PM, Peter S

Re: How to determine if repair need to be run

2011-03-31 Thread Peter Schuller
> Thanks a lot for elaborating on repairs.    Still, it's a bit fuzzy to me why > it is so important to run a repair before the GCGraceSeconds kicks in.   Does > this mean a delete does not get "replicated" ?   In other words when I delete > something on a node, doesn't cassandra set tombstones

Re: How to determine if repair need to be run

2011-03-31 Thread Jonathan Colby
Peter - Thanks a lot for elaborating on repairs.Still, it's a bit fuzzy to me why it is so important to run a repair before the GCGraceSeconds kicks in. Does this mean a delete does not get "replicated" ? In other words when I delete something on a node, doesn't cassandra set tombstones

Re: How to determine if repair need to be run

2011-03-31 Thread mcasandra
-incubator-apache-org.3065146.n2.nabble.com/How-to-determine-if-repair-need-to-be-run-tp6220005p6227778.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: How to determine if repair need to be run

2011-03-31 Thread Peter Schuller
> silly question, would every cassandra installation need to have manual > repairs done on it? > > It would seem cassandra's "read repair" and regular compaction would take > care of keeping the data clean. > > Am I missing something? See my previous posts in this thread for the distinct reasons

Re: How to determine if repair need to be run

2011-03-31 Thread Jonathan Colby
silly question, would every cassandra installation need to have manual repairs done on it? It would seem cassandra's "read repair" and regular compaction would take care of keeping the data clean. Am I missing something? On Mar 30, 2011, at 7:46 PM, Peter Schuller wrote: >> I just wanted t

Re: How to determine if repair need to be run

2011-03-30 Thread Peter Schuller
> I just wanted to chime in here and say some people NEVER run repair. Just so long as the OP is understanding that this implies taking an explicit decision to accept the "misbehavior" you will see as a result. I.e., the reason people survive not doing repairs in some cases is, as in your case, th

Re: How to determine if repair need to be run

2011-03-30 Thread Peter Schuller
> I concur.  JIRA time? https://issues.apache.org/jira/browse/CASSANDRA-2405 -- / Peter Schuller

Re: How to determine if repair need to be run

2011-03-30 Thread Edward Capriolo
On Wed, Mar 30, 2011 at 12:54 PM, Peter Schuller wrote: >> Note this script doesn't work if your repair takes hours, and in the >> middle of the repair cassandra was restarted, nodetool will exit and the >> flagfile will be updated.   Another case, if repair hangs, and day later >> cassandra is re

Re: How to determine if repair need to be run

2011-03-30 Thread Peter Schuller
> Note this script doesn't work if your repair takes hours, and in the > middle of the repair cassandra was restarted, nodetool will exit and the > flagfile will be updated.   Another case, if repair hangs, and day later > cassandra is restarted. This is why "set -e" is at the to and commented as

Re: How to determine if repair need to be run

2011-03-30 Thread Chris Burroughs
On 03/29/2011 01:18 PM, Peter Schuller wrote: > (What *would* be useful perhaps is to be able to ask a node for the > time of its most recently started repair, to facilitate easier > comparison with GCGraceSeconds for monitoring purposes.) I concur. JIRA time? (Perhaps keeping track of the same

Re: How to determine if repair need to be run

2011-03-30 Thread Karl Hiramoto
On 03/30/11 00:31, Peter Schuller wrote: > > set -e # important > touch /path/to/flagfile.tmp > nodetool -h localhost repair > mv /path/to/flagfile.tmp /path/to/flagfile > Note this script doesn't work if your repair takes hours, and in the middle of the repair cassandra was restarted, node

Re: How to determine if repair need to be run

2011-03-29 Thread Peter Schuller
> I think what I feel is that there is a need to know if repair is required > flag in order for team to manage the cluster. And again, repair is always required essentially. You should *always* run it within the necessary period as determined by GCGraceSeconds. > Atleast at minimum, Is there a fl

Re: How to determine if repair need to be run

2011-03-29 Thread mcasandra
.3065146.n2.nabble.com/How-to-determine-if-repair-need-to-be-run-tp6220005p6221157.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: How to determine if repair need to be run

2011-03-29 Thread Peter Schuller
> So from what I am understanding is that there is no need to monitor this and > no need to remember running repair? If that's the case then manual repair > wouldn't be needed ever, correct? No. See my next-to-last e-mail where I go through two reasons to run nodetool repair, of which (a) is absol

Re: How to determine if repair need to be run

2011-03-29 Thread mcasandra
bble.com/How-to-determine-if-repair-need-to-be-run-tp6220005p6221041.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: How to determine if repair need to be run

2011-03-29 Thread Peter Schuller
> Looks like you didn't get to see my updated post :) This is the scenario I > was referring to: I don't see what's different. If you write a QUORUM and read at QUORUM, your read is guaranteed to see a previous write, period. If that cannot be satisfied, the read will fail due to not being able to

Re: How to determine if repair need to be run

2011-03-29 Thread mcasandra
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/How-to-determine-if-repair-need-to-be-run-tp6220005p6220683.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: How to determine if repair need to be run

2011-03-29 Thread Peter Schuller
First some specifics: > I think my problem is that I don't want to remember to run read repair. I You are not expected to remember to do so manually. Typically periodic repairs would be automated in some fashion, such as by having a cron job on each node that starts the repair. Typically some kin

Re: How to determine if repair need to be run

2011-03-29 Thread mcasandra
ds might be ok for some but not for everyone where we want bring all nodes in sync ASAP. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/How-to-determine-if-repair-need-to-be-run-tp6220005p6220423.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: How to determine if repair need to be run

2011-03-29 Thread Peter Schuller
> Thanks! I was keeping the discussion simple. But you make my case stronger > that we need such monitoring since it looks like it should always be run but > we want to run it as soon as it is required. The way to deal with individual requests timing out or transient flapping, is to use a consiste

Re: How to determine if repair need to be run

2011-03-29 Thread mcasandra
/How-to-determine-if-repair-need-to-be-run-tp6220005p6220228.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: How to determine if repair need to be run

2011-03-29 Thread Peter Schuller
> Yes but that doesn't really provide the monitoring that will really be > helpful. If I don't realize it until 2 days then we potentially could be > returning inconsistent results or not have data sync for 2 days until repair > is run. It will be best to be able to monitor these things so that it

Re: How to determine if repair need to be run

2011-03-29 Thread mcasandra
it can be run as soon as it is required (eg node down). Have such monitoring will be helpful for operations team to monitor also who may not know all internals of cassandra. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/How-to-determine-if-repa

Re: How to determine if repair need to be run

2011-03-29 Thread Peter Schuller
> Is there a way to monitor and tell if one of the node require repair? For eg: > Node was down and came back up but in the meantime HH were dropped. Now > unless we are really careful in all the scenarios we wouldn't have any > problems :) but in general when things are going awry you might forget

How to determine if repair need to be run

2011-03-29 Thread mcasandra
about running repair or other commands until there is a customer impact. Is there a way to monitor and alert on such things like repair? -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/How-to-determine-if-repair-need-to-be-run-tp6220005p6220005