Re: Repairing question

[Javier Canillas]

Paulo,

Thanks for your explanation.

2017-06-22 17:30 GMT-03:00 Paulo Motta :

> > This attribute seems to be only modified when executing "nodetool repair
> [keyspace] [table]", but not when executing with other options like
> --in-local-dc or --pr.
>
> This is correct behavior because this metric actually represent the
> percentage of SSTables incrementally repaired - and marked as repaired
> - which doesn't happen when you execute a non-incremental repair
> (--full, --in-local-dc, --pr). When you perform a non-incremental
> repair data is repaired but not marked as repaired since this require
> anti-compaction to be run.
>
> Actually this "percent repaired" display name is a bit misleading,
> since it sounds like data needs to be repaired while you could be
> running non-incremental repairs and still have data 100% repaired, so
> we should probably open a ticket to rename that to "Percent
> incrementally repaired" or similar.
>
>
> 2017-06-22 14:38 GMT-05:00 Javier Canillas :
> > Hi,
> >
> > I have been thinking about scheduling a daily routine to force repairs
> on a
> > cluster to maintain its health.
> >
> > I saw that by running a nodetool tablestats [keyspace] there is an
> attribute
> > called "Percent repaired" that show the percentage of data repaired on
> the
> > each table.
> >
> > This attribute seems to be only modified when executing "nodetool repair
> > [keyspace] [table]", but not when executing with other options like
> > --in-local-dc or --pr.
> >
> > My main concern is about building the whole MERKLE tree for a big table.
> I
> > have also check to repair by token ranges, but this also seems not to
> modify
> > this attribute of the table.
> >
> > Is this an expected behavior? Or there is something missing on the code
> that
> > needs to be fixed?
> >
> > My "maintenance" script would be calling nodetool tablestats per each
> > keyspace that has replication_factor > 0 to check for the value of the
> > "Percent repaired" of each table and, in case it is below some
> threshold, I
> > would execute a repair on it.
> >
> > Any ideas?
> >
> > Thanks in advance.
> >
> > Javier.
>

CREATE INDEX without IF NOT EXISTS when snapshoting

[Javier Canillas]

Hi everyone,

I came across something that bothers me a lot. I'm using snapshots to
backup data from my Cassandra cluster in case something really bad happens
(like dropping a table or a keyspace).

Exercising the recovery actions from those backups, I discover that the
schema put on the file "schema.cql" as a result of the snapshot has the
"CREATE IF NOT EXISTS" for the table, but not for the indexes.

When restoring from snapshots, and relying on the execution of these
schemas to build up the table structure, everything seems fine for tables
without secondary indexes, but for the ones that make use of them, the
execution of these statements fail miserably.

Here I paste a generated schema.cql content for a table with indexes:

CREATE TABLE IF NOT EXISTS keyspace1.table1 (
id text PRIMARY KEY,
content text,
last_update_date date,
last_update_date_time timestamp)
WITH ID = f1045fc0-2f59-11e7-95ec-295c3c064920
AND bloom_filter_fp_chance = 0.01
AND dclocal_read_repair_chance = 0.1
AND crc_check_chance = 1.0
AND default_time_to_live = 864
AND gc_grace_seconds = 864000
AND min_index_interval = 128
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE'
AND caching = { 'keys': 'NONE', 'rows_per_partition': 'NONE' }
AND compaction = { 'max_threshold': '32', 'min_threshold': '4', 'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' }
AND compression = { 'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor' }
AND cdc = false
AND extensions = {  };
*CREATE INDEX table1_last_update_date_idx ON keyspace1.table1
(last_update_date);*

I think the last part should be:

*CREATE INDEX IF NOT EXISTS table1_last_update_date_idx ON keyspace1.table1
(last_update_date);*

Any ideas? Have you wrote this part of the snapshot behavior for a
particular reason I'm not seeing?

I'm willing to help on coding (as I have done before xD) if you consider
this a trivial bug, but something that should be address.

Kind regards,

Javier.

Re: CREATE INDEX without IF NOT EXISTS when snapshoting

[Javier Canillas]

Kurt,

Thanks for your response. Created this ticket
. Feel free to add
anything to it that seems legit.

Downloading Cassandra code right now.

Fix seems quite simple. Expect a pull-request soon xD

2017-10-03 20:19 GMT-03:00 kurt greaves :

> Certainly would make sense and should be trivial.  here
> 
>  is
> where you want to look. Just create a ticket for it and prod here for a
> reviewer once you've got a change.​
>

Re: CREATE INDEX without IF NOT EXISTS when snapshoting

[Javier Canillas]

Well,

Patches submitted for version 3.0, 3.11 and trunk (although 3.0 patch can
be applied to 3.11). Feel free to review and comment.

Thanks!

2017-10-04 16:41 GMT-03:00 Javier Canillas :

> Kurt,
>
> Thanks for your response. Created this ticket
> <https://issues.apache.org/jira/browse/CASSANDRA-13935>. Feel free to add
> anything to it that seems legit.
>
> Downloading Cassandra code right now.
>
> Fix seems quite simple. Expect a pull-request soon xD
>
> 2017-10-03 20:19 GMT-03:00 kurt greaves :
>
>> Certainly would make sense and should be trivial.  here
>> <https://github.com/apache/cassandra/blob/982ab93a2f8a0f5c56af9378f65d3e9e43b9/src/java/org/apache/cassandra/db/ColumnFamilyStoreCQLHelper.java#L251>
>>  is
>> where you want to look. Just create a ticket for it and prod here for a
>> reviewer once you've got a change.​
>>
>
>

Re: CREATE INDEX without IF NOT EXISTS when snapshoting

[Javier Canillas]

Guys,

Hope you are all doing fine. Is there any news about this trivial issue
<https://issues.apache.org/jira/browse/CASSANDRA-13935>? Are you
considering to include it in next release? Or there is something I should
do?

Thanks!

Javier.

2017-10-05 10:41 GMT-03:00 Javier Canillas :

> Well,
>
> Patches submitted for version 3.0, 3.11 and trunk (although 3.0 patch can
> be applied to 3.11). Feel free to review and comment.
>
> Thanks!
>
> 2017-10-04 16:41 GMT-03:00 Javier Canillas :
>
>> Kurt,
>>
>> Thanks for your response. Created this ticket
>> <https://issues.apache.org/jira/browse/CASSANDRA-13935>. Feel free to
>> add anything to it that seems legit.
>>
>> Downloading Cassandra code right now.
>>
>> Fix seems quite simple. Expect a pull-request soon xD
>>
>> 2017-10-03 20:19 GMT-03:00 kurt greaves :
>>
>>> Certainly would make sense and should be trivial.  here
>>> <https://github.com/apache/cassandra/blob/982ab93a2f8a0f5c56af9378f65d3e9e43b9/src/java/org/apache/cassandra/db/ColumnFamilyStoreCQLHelper.java#L251>
>>>  is
>>> where you want to look. Just create a ticket for it and prod here for a
>>> reviewer once you've got a change.​
>>>
>>
>>
>

Column TTL misbehavior?

[Javier Canillas]

I have been working on Aquiles for some time, and some of the people that is
actually using my client to consume Cassandra's services (lastest version on
Cassandra 0.7.0) has followed me an issue. Básically, they were trying to
submit a new column with a TTL value of 852282524, and everytime they tried
to insert int, the following Thrift error was going back to them:

Thrift.TApplicationException: Internal error processing insert
at Apache.Cassandra.Cassandra.Client.recv_insert() in
D:\apache-cassandra-0.7.0-beta3\interface\gen-csharp\Apache\Cassandra\Cassandra.cs:line
485
at Apache.Cassandra.Cassandra.Client.insert(Byte[] key, ColumnParent
column_parent, Column column, ConsistencyLevel consistency_level) in
D:\apache-cassandra-0.7.0-beta3\interface\gen-csharp\Apache\Cassandra\Cassandra.cs:line
463

And the log within Cassandra was:

ERROR [pool-1-thread-45] 2011-01-16 19:03:35,514 Cassandra.java (line 2960)
Internal error processing insert
java.lang.AssertionError: -2147462157
at org.apache.cassandra.db.ExpiringColumn.(ExpiringColumn.java:57)
at org.apache.cassandra.db.ExpiringColumn.(ExpiringColumn.java:50)
at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:167)
at org.apache.cassandra.db.RowMutation.add(RowMutation.java:151)
at
org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:343)
at
org.apache.cassandra.thrift.Cassandra$Processor$insert.process(Cassandra.java:2952)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

I know that a TTL value of 9864 days is way too long, but after stepping
into the code, I found out the following:

public ExpiringColumn(ByteBuffer name, ByteBuffer value, long timestamp, int
timeToLive)
{
  this(name, value, timestamp, timeToLive, (int)
(System.currentTimeMillis() / 1000) + timeToLive);
}

So, the constructor of ExpiringColumn was basically converting a long into
an int and then adding the TTL value for that column. this value is the
placed on localExpirationTime which is used to check to see if the Column is
mark for deletation. This last variable is defined as integer.

So, after doing some math and rechecking the code, I think I have come
across a serious problem which means that, as time passes, the maximum
posible TTL value that might be set to an Expiring column must be each
second lower than before.

This is a huge limitation, as for now (using System.currentTimeMillis() from
my machine) the maximum posible TTL value is 9861 days (851990400 seconds)
now, but tomorrow it will be 9860 days and so on. So each day we came closer
to this issue.

I think that a good solution is changing the localExpirationTime from int to
long and, avoid converting a long value to its integer representation.

Just give me your thoughts about this.

Javier Canillas

Re: Column TTL misbehavior?

[Javier Canillas]

Avoid this subject, giving it a second thought, There is no reason to put a
column a TTL value that goes past a year, and this problem will only appear
after the year 2038.

I need to stop drinking coffee.

On Wed, Jan 19, 2011 at 7:08 PM, Javier Canillas
wrote:

> I have been working on Aquiles for some time, and some of the people that
> is actually using my client to consume Cassandra's services (lastest
> version on Cassandra 0.7.0) has followed me an issue. Básically, they were
> trying to submit a new column with a TTL value of 852282524, and everytime
> they tried to insert int, the following Thrift error was going back to them:
>
> Thrift.TApplicationException: Internal error processing insert
> at Apache.Cassandra.Cassandra.Client.recv_insert() in
> D:\apache-cassandra-0.7.0-beta3\interface\gen-csharp\Apache\Cassandra\Cassandra.cs:line
> 485
> at Apache.Cassandra.Cassandra.Client.insert(Byte[] key, ColumnParent
> column_parent, Column column, ConsistencyLevel consistency_level) in
> D:\apache-cassandra-0.7.0-beta3\interface\gen-csharp\Apache\Cassandra\Cassandra.cs:line
> 463
>
> And the log within Cassandra was:
>
> ERROR [pool-1-thread-45] 2011-01-16 19:03:35,514 Cassandra.java (line 2960)
> Internal error processing insert
> java.lang.AssertionError: -2147462157
> at org.apache.cassandra.db.ExpiringColumn.(ExpiringColumn.java:57)
> at org.apache.cassandra.db.ExpiringColumn.(ExpiringColumn.java:50)
> at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:167)
> at org.apache.cassandra.db.RowMutation.add(RowMutation.java:151)
> at
> org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:343)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$insert.process(Cassandra.java:2952)
> at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> I know that a TTL value of 9864 days is way too long, but after stepping
> into the code, I found out the following:
>
> public ExpiringColumn(ByteBuffer name, ByteBuffer value, long timestamp,
> int timeToLive)
> {
>   this(name, value, timestamp, timeToLive, (int)
> (System.currentTimeMillis() / 1000) + timeToLive);
> }
>
> So, the constructor of ExpiringColumn was basically converting a long into
> an int and then adding the TTL value for that column. this value is the
> placed on localExpirationTime which is used to check to see if the Column is
> mark for deletation. This last variable is defined as integer.
>
> So, after doing some math and rechecking the code, I think I have come
> across a serious problem which means that, as time passes, the maximum
> posible TTL value that might be set to an Expiring column must be each
> second lower than before.
>
> This is a huge limitation, as for now (using System.currentTimeMillis()
> from my machine) the maximum posible TTL value is 9861 days (851990400
> seconds) now, but tomorrow it will be 9860 days and so on. So each day we
> came closer to this issue.
>
> I think that a good solution is changing the localExpirationTime from int
> to long and, avoid converting a long value to its integer representation.
>
> Just give me your thoughts about this.
>
> Javier Canillas
>
>
>
>
>
>
>