unrecognized column family in logs

Hello, We run Cassandra 2.1.13 and since last few days we're seeing below in logs occasionally. The node then becomes unresponsive to cqlsh. ERROR [SharedPool-Worker-2] 2017-11-08 17:02:32,362 CommitLogSegment.java:441 - Attempted to write commit log entry for unrecognized column family: 2854d

RE: Re: Tuning bootstrap new node

You can change YAML setting of memtable_cleanup_threshold to 0.7 (from the default of 0.3). This will push SSTables to disk less often and will reduce the compaction time. While this won’t change the streaming time, it will reduce the overall time for your node to be healthy. From: Harikrishna

RE: Cassandra proxy to control read/write throughput

There are some caveats with coordinator only nodes. You can read about our experience in detail here. From: Nate McCall [mailto:n...@thelastpickle.com] Sent: Sunday, October 29, 2017 2:12 PM To: Cassandra Us

RE: nodetool repair failure

If possible, simply read the table under question with consistency=ALL. This will trigger a repair and is far more reliable than the nodetool command. From: Balaji Venkatesan [mailto:venkatesan.bal...@gmail.com] Sent: Thursday, June 29, 2017 7:26 PM To: user@cassandra.apache.org Subject: Re: node

Cassandra 2.1.13: Using JOIN_RING=False

Hello, With some inspiration from the Cassandra Summit talk from last year, we are trying to setup a cluster with coordinator-only nodes. We setup join_ring=false in env.sh, disabled auth in YAML and the nodes are able to start just fine. However, we're running into a few problems 1] The nodes

RE: RemoveNode CPU Spike Question

Well, looking through logs I confirmed that my understanding below is correct, but would be good to hear from experts for sure 😊 From: Anubhav Kale [mailto:anubhav.k...@microsoft.com] Sent: Tuesday, January 10, 2017 9:58 AM To: user@cassandra.apache.org Cc: Sean Usher Subject: RemoveNode CPU

RemoveNode CPU Spike Question

Hello, Recently, I started noticing an interesting pattern. When I execute "removenode", a subset of the nodes that now own the tokens result it in a CPU spike / disk activity, and sometimes SSTables on those nodes shoot up. After looking through the code, it appears to me that below function f

RE: High CPU on nodes

U%3D&reserved=0> 2016-12-17 0:10 GMT+01:00 Anubhav Kale mailto:anubhav.k...@microsoft.com>>: Hello, I am trying to fight a high CPU problem on some of our nodes. Thread dumps show that it’s not GC threads (we have 30GB heap), iostat %iowait confirms it’s not disk (ranges

High CPU on nodes

Hello, I am trying to fight a high CPU problem on some of our nodes. Thread dumps show that it's not GC threads (we have 30GB heap), iostat %iowait confirms it's not disk (ranges between 0.3 - 0.9%). One of the ways in which the problem manifests is that the nodes can't compact SSTables and it

RE: Question on Read Repair

Subject: Re: Question on Read Repair Yes: https://github.com/apache/cassandra/blob/81f6c784ce967fadb6ed7f58de1328e713eaf53c/src/java/org/apache/cassandra/db/ConsistencyLevel.java#L286 From: Anubhav Kale mailto:anubhav.k...@microsoft.com>> Reply-To: "user@cassandra.apache.org&

RE: Backup restore with a different name

You would have to build some logic on top of what’s natively supported. Here is an option: https://github.com/anubhavkale/CassandraTools/tree/master/BackupRestore From: Jens Rantil [mailto:jens.ran...@tink.se] Sent: Wednesday, November 2, 2016 2:21 PM To: Cassandra Group Subject: Backup restor

Rebuilding with vnodes

Hello, I am trying to rebuild a new Data Center with 50 Nodes, and expect 1 TB / node. Nodes are backed by SSDs, and the rebuild is happening from another DC in same physical region. This is with 2.1.13. I am doing this with stream_throughput=200 MB, concurrent_compactors=256, compactionthroug

Anticompaction Question

Hello, If incremental repairs are enabled, there is logic in every compaction strategy to make sure not to mix repaired and unrepaired SS Tables. Does this mean if some SS Table files are repaired and some aren't and incremental repairs don't work reliably, the unrepaired tables will never get

New node overstreaming data ?

Hello, We run 2.1.13 and seeing an odd issue. A node went down, and stayed down for a while so it went out of gossip. When we try to bootstrap it again (as a new node), it overstreams from other nodes, eventually disk becomes full and crashes. This repeated 3 times. Does anyone have any insigh

RE: Repair in Multi Datacenter - Should you use -dc Datacenter repair or repair with -pr

repaired (it’s a simpler version of mutation based repair in a sense), so Cassandra has no way to prove that the data has been repaired. With tickets like https://issues.apache.org/jira/browse/CASSANDRA-6434, this has implications on tombstone removal. From: Anubhav Kale mailto:anubhav.k

RE: Repair in Multi Datacenter - Should you use -dc Datacenter repair or repair with -pr

The default repair process doesn't usually work at scale, unfortunately. Depending on your data size, you have the following options. Netflix Tickler: https://github.com/ckalantzis/cassTickler (Read at CL.ALL via CQL continuously :: Python) Spotify Reaper: https://github.com/spotify/cassandra-

VNode Streaming Math

Hello, Suppose I have a 100 node ring, with num_tokens=32 (thus, 32 VNodes per physical machine). Assume this cluster has just one keyspace having one table. There are 10 SS Tables on each node, and size on disk is 10GB on each node. For simplicity, assume each SSTable is 1GB. Now, a node went

RE: Question on Read Repair

node is down, it won’t attempt a read, because the consistency level can’t be satisfied – none of the other replicas will be repaired. From: Anubhav Kale mailto:anubhav.k...@microsoft.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org&

Question on Read Repair

Hello, This is more of a theory / concept question. I set CL=ALL and do a read. Say one replica was down, will the rest of the replicas get repaired as part of this ? (I am hoping the answer is yes). Thanks !

RE: Nodetool rebuild question

ect: Re: Nodetool rebuild question If you set RF to 0, you can ignore my second sentence/paragraph. The third still applies. From: Anubhav Kale mailto:anubhav.k...@microsoft.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@ca

RE: Nodetool rebuild question

~100 more nodes into a cluster, you’d have to wait potentially a day or more per node to compact away the leftovers before bootstrapping the next, which is prohibitive at scale. - Jeff From: Anubhav Kale mailto:anubhav.k...@microsoft.com>> Reply-To: "user@cassandra

Nodetool rebuild question

Hello, As part of rebuild, I noticed that the destination node gets -tmp- files from other nodes. Are following statements correct ? 1. The files are written to disk without going through memtables. 2. Regular compactors eventually compact them to bring down # SSTables to a reason

RE: Repairs at scale in Cassandra 2.1.13

- Apache Cassandra Consulting http://www.thelastpickle.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.thelastpickle.com&data=01%7C01%7CAnubhav.Kale%40microsoft.com%7C698bf80ea0aa4b86e85608d3e79938db%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=q9UnaVZgS0HekWDbw

Repairs at scale in Cassandra 2.1.13

Hello, We run Cassandra 2.1.13 (don't have plans to upgrade yet). What is the best way to run repairs at scale (400 nodes, each holding ~600GB) that actually works ? I'm considering doing subrange repairs (https://github.com/BrianGallew/cassandra_range_repair/blob/master/src/range_repair.py) a

RE: Token Ring Question

is correct, In my view, unless the drivers execute the *Topology.GetReplicas from Cassandra core somehow (something that isn’t available to them), they will never be able to tell the correct node that holds data for a given token. Is my understanding wrong ? From: Anubhav Kale [mailto:anubhav.k

RE: StreamCoordinator.ConnectionsPerHost set to 1

CASSANDRA-12015. 2016-06-16 14:43 GMT-03:00 Anubhav Kale mailto:anubhav.k...@microsoft.com>>: Hello, I noticed that StreamCoordinator.ConnectionsPerHost is always set to 1 (Cassandra 2.1.13). If I am reading the code correctly, this means there will always be just one socket (well, 2 technical

StreamCoordinator.ConnectionsPerHost set to 1

Hello, I noticed that StreamCoordinator.ConnectionsPerHost is always set to 1 (Cassandra 2.1.13). If I am reading the code correctly, this means there will always be just one socket (well, 2 technically for each direction) between nodes when rebuilding thus the data will always be serialized.

RE: Token Ring Question

idea for some tools. In reality, it's just the replica that owns the first (clockwise) token. I'm not sure what you're really asking, though -- what are you concerned about? On Wed, Jun 1, 2016 at 2:40 PM, Anubhav Kale mailto:anubhav.k...@microsoft.com>> wrote: Hello,

Token Ring Question

Hello, I recently learnt that regardless of number of Data Centers, there is really only one token ring across all nodes. (I was under the impression that there is one per DC like how Datastax Ops Center would show it). Suppose we have 4 v-nodes, and 2 DCs (2 nodes in each DC) and a key space i

RE: Removing a datacenter

keyspace, remove the replication settings for that DC, and then you can decommission (and they won’t need to stream nearly as much, since they no longer own that data – decom will go much faster). From: Anubhav Kale Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.or

Removing a datacenter

Hello, Suppose we have 2 DCs and we know that the data is correctly replicated in both. In such situation, is it safe to "remove" one of the DCs by simply doing a "nodetool remove node" followed by "nodetool removenode force" for each node in that DC (instead of doing a "nodetool decommission"

Applying TTL Change quickly

Hello, We use STCS and DTCS on our tables and recently made a TTL change (reduced from 8 days to 2) on a table with large amounts of data. What is the best way to quickly purge old data ? I am playing with tombstone_compaction_interval at the moment, but would like some suggestions on what else

Nodetool clearsnapshot doesn't support Column Families

Hello, I noticed that clearsnapshot doesn't support removing snapshots on a per CF, like how snapshots lets you take it per CF. http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsClearSnapShot.html I couldn't find a JIRA to address this. Is this intentional ? If so, I am curious to

Nodetool repair question

Hello, Suppose I have 3 nodes, and stop Cassandra on one of them. Then I run a repair. Will repair move the token ranges from down node to other node ? In other words in any situation, does repair operation ever change token ownership ? Thanks !

RE: SS Tables Files Streaming

@cassandra.apache.org<mailto:user@cassandra.apache.org>" Subject: Re: SS Tables Files Streaming Repairs, bootstamp, decommission. On Fri, May 6, 2016 at 1:16 PM Anubhav Kale mailto:anubhav.k...@microsoft.com>> wrote: Hello, In what scenarios can SS Table files on disk from Node 1 go to N

SS Tables Files Streaming

Hello, In what scenarios can SS Table files on disk from Node 1 go to Node 2 as is ? I'm aware this happens in nodetool rebuild and I am assuming this does not happen in repairs. Can someone confirm ? The reason I ask is I am working on a solution for backup / restore and I need to be sure if

SS Table File Names not containing GUIDs

Hello, I am wondering if there is any reason as to why the SS Table format doesn't have a GUID. As far as I can tell, the incrementing number isn't really used for any special purpose in code, and having a unique name for the file seems to be a better thing, in general. Specifically, this caus

RE: Problem Replacing a Dead Node

Reusing the bootstrapping node could have caused this, but hard to tell. Since you have only 7 nodes, have you tried doing a few rolling restarts of all nodes to let gossip settle ? Also, the node is pingable from other nodes even though it says Unreachable below. Correct ? Based on nodetool st

RE: Problem Replacing a Dead Node

Is the datastax-agent running fine on the node ? What does nodetool status and system.log show ? From: Mir Tanvir Hossain [mailto:mir.tanvir.hoss...@gmail.com] Sent: Thursday, April 21, 2016 10:02 AM To: user@cassandra.apache.org Subject: Problem Replacing a Dead Node Hi, I am trying to replace

Changing Racks of Nodes

Hello, If a running node moves around and changes its rack in the process, when its back in the cluster (through ignore-rack property), is it a correct statement that queries will not see some data residing on this node until a repair is run ? Or, is it more like the node may get requests for

RE: Nodetool rebuild and bootstrap

: Nodetool rebuild and bootstrap https://issues.apache.org/jira/browse/CASSANDRA-8838 Bootstrap only resumes on 2.2.0 and newer. I’m unsure of rebuild, but I suspect it does not resume at all. From: Anubhav Kale Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"

Nodetool rebuild and bootstrap

Hello, Is it a correct statement that both rebuild and bootstrap resume streaming from where they were left off (meaning they don't stream the entire data again) in case of node restarting during rebuild / bootstrap process ? Thanks !

RE: Leak Detected while bootstrap

on that ticket and share your logs leading up to the error? On Wed, Apr 13, 2016 at 3:37 PM, Anubhav Kale mailto:anubhav.k...@microsoft.com>> wrote: Hello, Since we upgraded to Cassandra 2.1.12, we are noticing that below happens when we are trying to bootstrap nodes, and the process just ge

Leak Detected while bootstrap

Hello, Since we upgraded to Cassandra 2.1.12, we are noticing that below happens when we are trying to bootstrap nodes, and the process just gets stuck. Restarting the process / VM does not help. Our nodes are around ~300 GB and run on local SSDs and we haven't seen this problem on older versio

RE: Removing a DC

that could be affected by this issue? Best, Joel On Thu, Apr 7, 2016 at 11:51 AM, Anubhav Kale mailto:anubhav.k...@microsoft.com>> wrote: Hello, We removed a DC using instructions from https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_decomission_dc

Removing a DC

Hello, We removed a DC using instructions from https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_decomission_dc_t.html After all nodes were gone, 1. System.peers don't have an entry for the nodes that were removed. (confirmed via a cqlsh query with consistency all) 2.

RE: Speeding up "nodetool rebuild"

icrosoft.com%7c83413eba6d6a41c15a4608d35a0b5741%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=nt4LvhgsIHTb2IyfYGwDSy3%2biaTslE4%2ffNLfGTgAExs%3d> 2016-03-31 23:19 GMT+02:00 Anubhav Kale mailto:anubhav.k...@microsoft.com>>: Thanks, is there any way to determine that rebuild is complete. Based on following line in

RE: Speeding up "nodetool rebuild"

r not) isRebuilding.set(false); } -Original Message- From: Eric Evans [mailto:eev...@wikimedia.org] Sent: Thursday, March 31, 2016 9:50 AM To: user@cassandra.apache.org Subject: Re: Speeding up "nodetool rebuild" On Wed, Mar 30, 2016 at 3:44 PM, Anubhav Kale wro

Speeding up "nodetool rebuild"

Hello, Will changing compactionthroughput and streamingthroughput help with reducing the "rebuild" time on a brand new node ? We will do it both on the new node, and the nodes in source DC from where data is streamed. Any other ways to make the "rebuild" faster ? Thanks !

RE: Rack aware question.

ven when a rack-level failure occurs. In short, if CL=ALL is acceptable, then you might as well dump the rack-aware approach, which was how you got into this situation in the first place. -- Jack Krupansky On Wed, Mar 23, 2016 at 7:31 PM, Anubhav Kale mailto:anubhav.k...@microsoft.com>&

RE: Rack aware question.

consistency to ALL, and now I can get data all the time. From: Anubhav Kale [mailto:anubhav.k...@microsoft.com] Sent: Wednesday, March 23, 2016 4:14 PM To: user@cassandra.apache.org Subject: RE: Rack aware question. Thanks, Read repair is what I thought must be causing this, so I experimented some

RE: Rack aware question.

ust be deprecated. ignore_rack flag can be useful if you move your data manually, with rsync or sstableloader. 2016-03-23 19:09 GMT-03:00 Anubhav Kale mailto:anubhav.k...@microsoft.com>>: Thanks for the pointer – appreciate it. My test is on the latest trunk and slightly different. I am

RE: Rack aware question.

oft.com%7c7741553cdb7c4ce7ee1f08d3536599a0%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3PY62w9X94T3fCkPZVJzN2dl8eda44Yj3zBvk83faWk%3d> =Rob On Wed, Mar 23, 2016 at 2:06 PM, Anubhav Kale mailto:anubhav.k...@microsoft.com>> wrote: Oh, and the query I ran was “select * from racktest.racktable where id=1”

RE: Rack aware question.

Oh, and the query I ran was “select * from racktest.racktable where id=1” From: Anubhav Kale [mailto:anubhav.k...@microsoft.com] Sent: Wednesday, March 23, 2016 2:04 PM To: user@cassandra.apache.org Subject: RE: Rack aware question. Thanks. To test what happens when rack of a node changes in a

Rack aware question.

Hello, Suppose we change the racks on VMs on a running cluster. (We need to do this while running on Azure, because sometimes when the VM gets moved its rack changes). In this situation, new writes will be laid out based on new rack info on appropriate replicas. What happens for existing data

RE: Expiring Columns

I think the answer is no. There are explicit checks in Read code path to ignore anything that’s past the TTL (based on local time of the node under question). From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in] Sent: Monday, March 21, 2016 5:19 AM To: User Subject: Expiring Columns Hi, I want t

DTCS Question

I am using Cassandra 2.1.13 which has all the latest DTCS fixes (it does STCS within the DTCS windows). It also introduced a field called MAX_WINDOW_SIZE which defaults to one day. So in my data folders, I may see SS Tables that span beyond a day (generated through old data through repairs or c

RE: DTCS bucketing Question

wordpress.com/2014/12/dtcs3.png [Anubhav Kale] This doesn’t seem correct. In the original test (look at comments), the first window is pretty big and in many cases, the first window is big. > Note, that if I keep the base to original (100L) or increase it and play with > min_threshold the

DTCS bucketing Question

Hello, I am trying to concretely understand how DTCS makes buckets and I am looking at the DateTieredCompactionStrategyTest.testGetBuckets method and played with some of the parameters to GetBuckets method call (Cassandra 2.1.12). I don't think I fully understand something there. Let me try t

RE: DTCS Question

Thanks for the explanation. From: Marcus Eriksson [mailto:krum...@gmail.com] Sent: Thursday, March 17, 2016 12:56 AM To: user@cassandra.apache.org Subject: Re: DTCS Question On Wed, Mar 16, 2016 at 6:49 PM, Anubhav Kale mailto:anubhav.k...@microsoft.com>> wrote: I am using Cassandra