Re: Deletes Reappeared even when nodes are not down

2015-11-16 Thread Robert Coli
On Sat, Nov 14, 2015 at 9:58 AM, Peddi, Praveen wrote: > I checked tpstats and there are no dropped mutations (though I checked it > after restating the affected nodes). If the problem occurs again, I will > check tpstats again. Is there any stat that shows failed hints? The only > abnormality I

Re: Deletes Reappeared even when nodes are not down

2015-11-14 Thread Peddi, Praveen
@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: Deletes Reappeared even when nodes are not down On Fri, Nov 13, 2015 at 1:47 PM, Peddi, Praveen mailto:pe...@amazon.com>> wrote: We do not currently run repairs because we know our deployment time for each

Re: Deletes Reappeared even when nodes are not down

2015-11-13 Thread Robert Coli
On Fri, Nov 13, 2015 at 1:47 PM, Peddi, Praveen wrote: > We do not currently run repairs because we know our deployment time for > each cassandra node is very short. I do understand we have to run repairs > but would repair be in the picture here when no nodes in the cluster were > down for last

Re: Deletes Reappeared even when nodes are not down

2015-11-13 Thread Jonathan Haddad
andra.apache.org" > Subject: Re: Deletes Reappeared even when nodes are not down > > On Fri, Nov 13, 2015 at 1:09 PM, Peddi, Praveen wrote: > >> We are seeing a scenario where some of the rows in the table reappears >> even after they are deleted. We have seen this i

Re: Deletes Reappeared even when nodes are not down

2015-11-13 Thread Peddi, Praveen
@cassandra.apache.org>> Subject: Re: Deletes Reappeared even when nodes are not down On Fri, Nov 13, 2015 at 1:09 PM, Peddi, Praveen mailto:pe...@amazon.com>> wrote: We are seeing a scenario where some of the rows in the table reappears even after they are deleted. We have seen this in Pro

Re: Deletes Reappeared even when nodes are not down

2015-11-13 Thread Robert Coli
On Fri, Nov 13, 2015 at 1:09 PM, Peddi, Praveen wrote: > We are seeing a scenario where some of the rows in the table reappears > even after they are deleted. We have seen this in Prod 3 times in last 1 > week and *coincidentally all 3 times on the same partition*. We have > confirmed that nodes

Re: Deletes Reappeared even when nodes are not down

2015-11-13 Thread Jon Haddad
> To: "user@cassandra.apache.org <mailto:user@cassandra.apache.org>" > mailto:user@cassandra.apache.org>> > Subject: Re: Deletes Reappeared even when nodes are not down > > Well, yeah, it's still possible. Are your clocks 20 minutes off? > >> On No

Re: Deletes Reappeared even when nodes are not down

2015-11-13 Thread Jon Haddad
user@cassandra.apache.org>> > Date: Friday, November 13, 2015 at 4:13 PM > To: "user@cassandra.apache.org <mailto:user@cassandra.apache.org>" > mailto:user@cassandra.apache.org>> > Subject: Re: Deletes Reappeared even when nodes are not down > > Any

Re: Deletes Reappeared even when nodes are not down

2015-11-13 Thread Peddi, Praveen
3, 2015 at 4:24 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: Deletes Reappeared even when nodes are not down Well, yeah, it's still possible. Are your clocks 20 minutes off? On Nov 13, 2015, at 1:20 PM, P

Re: Deletes Reappeared even when nodes are not down

2015-11-13 Thread Peddi, Praveen
>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Date: Friday, November 13, 2015 at 4:13 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org

Re: Deletes Reappeared even when nodes are not down

2015-11-13 Thread Jon Haddad
Any chance your clocks are off? > On Nov 13, 2015, at 1:09 PM, Peddi, Praveen wrote: > > Hi, > We are using Cassandra 2.0.8, with replication factor of 3. > > We are seeing a scenario where some of the rows in the table reappears even > after they are deleted. We have seen this in Prod 3 tim

Deletes Reappeared even when nodes are not down

2015-11-13 Thread Peddi, Praveen
Hi, We are using Cassandra 2.0.8, with replication factor of 3. We are seeing a scenario where some of the rows in the table reappears even after they are deleted. We have seen this in Prod 3 times in last 1 week and coincidentally all 3 times on the same partition. We have confirmed that nodes