Re: Repair does not fix inconsistency

This looks to me more like a secondary index issue. If you say the access via rowkey is always correct, then the repair works fine. I think there might be something wrong with your secondary index then. Just a follow up in case someone will have the same case: This problem was solved by runni

Re: Repair does not fix inconsistency

Little update ;-) It couldn't be so easy - I can't drop these indexes :P 1) cqlsh: cqlsh:production> DROP INDEX Users_ Users_active_idxUsers_email_idx Users_username_idx cqlsh:production> DROP INDEX Users_email_idx ; cqlsh:production> DROP INDEX Users_ Users_active_idxUsers_email_id

Re: Repair does not fix inconsistency

W dniu 04.04.2013 15:38, horschi pisze: I'm glad to hear that. I feared my ticket might be responsible for your data loss. I could not live the guilt ;-) Seriously: I'm glad we can rule out the repair change. Haha, I didn't notice before that it was your ticket! ;-) Yes, if it works with CL=o

Re: Repair does not fix inconsistency

Yes, maybe there are two issues here: repair not running and maybe really some index-thing. Repair is fine - all the data seem to be in SSTables. I've checked it and while index tells me that I have 1 tombstone and 0 live cells for a key, I can _see_, thanks to sstable2json, that I have 3 "l

Re: Repair does not fix inconsistency

Repair is fine - all the data seem to be in SSTables. I've checked it and > while index tells me that I have 1 tombstone and 0 live cells for a key, I > can _see_, thanks to sstable2json, that I have 3 "live cells" (assuming a > cell is an entry in SSTable) and 0 tombstones. After being confused fo

Re: Repair does not fix inconsistency

> Well... Strange. We have such problem with 6 users, but there's only ONE > tombstone (created 8 days ago, so it's not gcable yet) in all the SSTables > on 2:1 node - checked using sstable2json. > Moreover, this tombstone DOES NOT belong to the row key I'm using for > tests, because this user was

Re: Repair does not fix inconsistency

Does CQL not allow CL=ONE queries? Why does it ask two nodes for the key, when you say that you are using CL=default=1? I'm a bit confused here (I'm a thrift user). Yup, that's another thing I'm curious about too (default CL is ONE for sure), but as for now it helps me to investigate my probl

Re: Repair does not fix inconsistency

Well... Strange. We have such problem with 6 users, but there's only ONE tombstone (created 8 days ago, so it's not gcable yet) in all the SSTables on 2:1 node - checked using sstable2json. Moreover, this tombstone DOES NOT belong to the row key I'm using for tests, because this user was NOT eve

Re: Repair does not fix inconsistency

Hi, This was my first thought too, but if you take a look at the logs I > attached to previous e-mail, you'll notice that query "by key" > (no-index.log) retrieves data from BOTH replicas, while the "by indexed > column" one (index.log) talks only to one of them (too bad it's the one > that contai

Re: Repair does not fix inconsistency

Hi Sylvain, Thanks for explaination :-) However, in this case, I still do not get why this (probably) gcable tombstone on 2:1 could cause this mess. As AE ignores only the tombstone itself (which means that there are no data for this key on 2:1 node from repair's point of view), it should resu

Re: Repair does not fix inconsistency

Hi Christian, About CASSANDRA-4905 - thanks for explaining this :-) This looks to me more like a secondary index issue. If you say the access via rowkey is always correct, then the repair works fine. I think there might be something wrong with your secondary index then. This was my first thou

Re: Repair does not fix inconsistency

Hi Michal, Let's say the tombstone on one of the nodes (X) is gcable and was not > compacted (purged) so far. After it was created we re-created this row, but > due some problems it was written only to the second node (Y), so we have > "live" data on node Y which is newer than the gcable tombstone

Re: Repair does not fix inconsistency

> I'm considering a problem related to this issue: > https://issues.apache.org/**jira/browse/CASSANDRA-4905 > > Let's say the tombstone on one of the nodes (X) is gcable and was not > compacted (purged) so far. After it was created we re-created

Re: Repair does not fix inconsistency

Hi Aaron, At first, before I go with a lot of logs: I'm considering a problem related to this issue: https://issues.apache.org/jira/browse/CASSANDRA-4905 Let's say the tombstone on one of the nodes (X) is gcable and was not compacted (purged) so far. After it was created we re-created this r

Re: Repair does not fix inconsistency

What version are you on ? Can you run a repair on the CF and check: Does the repair detect differences in the CF and stream changes ? After the streaming does it run a secondary index rebuild on the new sstable ? (Should be in the logs) Can you provide the full query trace ? Cheers ---

Repair does not fix inconsistency

Hi, TL;DR: I have inconsistend data (1 live row on node A & 1 tombstoned row on node B) that do not get fixed by repair. What can be a problem? Long version: I have a CF containing Users' info, which I sometimes query by key, and sometimes by indexed columns like email. I'm using RF=2. I wri