DTCS Question

[Anubhav Kale]

Hello,

I am trying to concretely understand how DTCS makes buckets and I am looking at 
the DateTieredCompactionStrategyTest.testGetBuckets method and played with some 
of the parameters to GetBuckets method call (Cassandra 2.1.12).

I don't think I fully understand something there. Let me try to explain.

Consider the second test there. I changed the pairs a bit for easier 
explanation and changed base (initial window size)=1000L and Min_Threshold=2

pairs = Lists.newArrayList(
Pair.create("a", 200L),
Pair.create("b", 2000L),
Pair.create("c", 3600L),
Pair.create("d", 3899L),
Pair.create("e", 3900L),
Pair.create("f", 3950L),
Pair.create("too new", 4125L)
);
buckets = getBuckets(pairs, 1000L, 2, 4050L, Long.MAX_VALUE);

In this case, the buckets should look like [0-4000] [4000-]. Is this correct ? 
The buckets that I get back are different ("a" lives in its bucket and everyone 
else in another). What I am missing here ?

Another case,

pairs = Lists.newArrayList(
Pair.create("a", 200L),
Pair.create("b", 2000L),
Pair.create("c", 3600L),
Pair.create("d", 3899L),
Pair.create("e", 3900L),
Pair.create("f", 3950L),
Pair.create("too new", 4125L)
);
buckets = getBuckets(pairs, 50L, 4, 4050L, Long.MAX_VALUE);

Here, the buckets should be [0-3200] [3200-4000] [4000-4050] [4050-]. Is this 
correct ? Again, the buckets that come back are quite different.

Note, that if I keep the base to original (100L) or increase it and play with 
min_threshold the results are exactly what I would expect.

The way I think about DTCS is, try to make buckets of maximum possible sizes 
from 0, and once you can't make do that , make smaller buckets (similar to what 
the comment suggests). Is this mental model wrong ? I am afraid that the math 
in Target class is somewhat hard to follow so I am thinking about it this way.

Thanks a lot in advance.

-Anubhav

RE: DTCS Question

[Anubhav Kale]

Thanks for the long explanation, I looked at the link you pointed to and it 
does seem to concur with my mental model. 

Do you see any issues with that model and simplify this logic to 

1. Create windows from start (min) to end (max) going from maximum possible 
size.
2. Scan all SS Tables and put them in appropriate buckets.

To be honest, the Target class is really difficult to reason about. The reason 
I investigated this was we wanted to reason about how our SS Tables are 
looking, and I unfortunately can't.

Thanks again for the explanation !!

-Original Message-
From: Björn Hegerfors [mailto:bj...@spotify.com] 
Sent: Thursday, March 17, 2016 11:19 AM
To: dev@cassandra.apache.org
Subject: Re: DTCS Question

That is probably close to the actual way it works, but not quite equal. My 
mental model when making this went backwards in time, towards 0, not forwards.

It's something like this (using the numbers from your first example): make a 
bucket of the specified "timeUnit" size (1000), that contains the "now"
timestamp (4050), where the starting (and therefore also the ending) timestamp 
of the bucket is 0 modulo the size of the bucket. That last point is perhaps 
the trickiest point to follow. There is only one such place for the bucket, 
[4000-5000) in this case. No other bucket that is aligned with the 1000s can 
contain 4050.

Now, the next bucket (backwards) is computed based on this [4000-5000) bucket. 
Most of the time it will simply be the same-sized bucket right before it, i.e. 
[3000-4000), but if the start timestamp of our bucket (4000), divided by its 
size (so 4), is 0 modulo "base" (2 in this case), which it happens to be here, 
then we increase out bucket size "base" times, and instead make the bucket of 
*that* size that ends right before our current bucket. So the result will be 
[2000-4000).

This method of getting the next bucket is repeated until we reach timestamp 0. 
Using the above logic, we don't increase the size of the bucket this time, 
because we have a start timestamp of 2000 which becomes 1 when divided by the 
size (2000). So we end up with [0, 2000), and we're done.
The buckets were [4000-5000), [2000-4000) and [0-2000).

What's more important than understanding these rules is of course getting some 
kind of intuition for this. Here's what it boils down to: we want there to be 
"base" equally sized buckets right next to each other before we
*coalesce* them. Every bucket is aligned with its own size (as an analogy, 
compilers typically align 4-byte integers on addresses divisible by 4, same 
concept). So, by extension, the bigger bucket they coalesce into must be 
aligned with *its* size. Not just any "base" adjacent buckets will do, it will 
be those that align with the next size.

The remaining question is when do they coalesce? There will always be at least 
1 and at most "base" buckets of every size. Say "base"=4, then there can be 4 
bucket of some size (by necessity next to each other and aligned on 4 times 
their size). The moment a new bucket of the same size appears, the 4 buckets 
become one and this "fifth" bucket will be alone with its size (and the start 
of a new group of 4 such buckets). (The rule for making the bucket were the 
"now" timestamp lives in, is where new buckets come from).

I wish this was easier to explain in simple terms. I personally find this to 
have very nice properties, in that it gives every bucket a fair amount of time 
to settle before it's time for the next compaction.

Interestingly, I proposed an alternative algorithm in this ticket 
<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fCASSANDRA-9013&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7ca41a30e8a81144de0cd008d34e90a502%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=QvmtOPUWqYnoFLLY0jx7%2fqgg0tJSleAsdBzo%2bzh22G4%3d>,
 including a patch implementing it. My gut tells me that the mental model that 
you've used here is actually equivalent to that algorithm in the ticket. It's 
just expressed in a very different way. Might be something for me to try to 
prove when I'm really bored :)

Hope this helped! Any particular reason you're investigating this?

/
Bj0rn

On Thu, Mar 17, 2016 at 5:43 PM, Anubhav Kale 
wrote:

> Hello,
>
> I am trying to concretely understand how DTCS makes buckets and I am 
> looking at the DateTieredCompactionStrategyTest.testGetBuckets method 
> and played with some of the parameters to GetBuckets method call 
> (Cassandra 2.1.12).
>
> I don't think I fully understand something there. Let me try to explain.
>
> Consider the second test there. I changed the pairs a bit for easier 
> explanation and changed base (initial window size)=1000L and 
> Min_Threshold=2
>
> pairs = Lists.n

RE: DTCS Question

[Anubhav Kale]

I tried the patch, and while it works almost similar to what I described below, 
it does deviate at times due to "alignment" logic.

I opened https://issues.apache.org/jira/browse/CASSANDRA-11407 and provided a 
patch there. I would appreciate thoughts. 

-Original Message-----
From: Anubhav Kale [mailto:anubhav.k...@microsoft.com] 
Sent: Thursday, March 17, 2016 12:08 PM
To: dev@cassandra.apache.org
Subject: RE: DTCS Question

Thanks for the long explanation, I looked at the link you pointed to and it 
does seem to concur with my mental model. 

Do you see any issues with that model and simplify this logic to 

1. Create windows from start (min) to end (max) going from maximum possible 
size.
2. Scan all SS Tables and put them in appropriate buckets.

To be honest, the Target class is really difficult to reason about. The reason 
I investigated this was we wanted to reason about how our SS Tables are 
looking, and I unfortunately can't.

Thanks again for the explanation !!

-Original Message-
From: Björn Hegerfors [mailto:bj...@spotify.com]
Sent: Thursday, March 17, 2016 11:19 AM
To: dev@cassandra.apache.org
Subject: Re: DTCS Question

That is probably close to the actual way it works, but not quite equal. My 
mental model when making this went backwards in time, towards 0, not forwards.

It's something like this (using the numbers from your first example): make a 
bucket of the specified "timeUnit" size (1000), that contains the "now"
timestamp (4050), where the starting (and therefore also the ending) timestamp 
of the bucket is 0 modulo the size of the bucket. That last point is perhaps 
the trickiest point to follow. There is only one such place for the bucket, 
[4000-5000) in this case. No other bucket that is aligned with the 1000s can 
contain 4050.

Now, the next bucket (backwards) is computed based on this [4000-5000) bucket. 
Most of the time it will simply be the same-sized bucket right before it, i.e. 
[3000-4000), but if the start timestamp of our bucket (4000), divided by its 
size (so 4), is 0 modulo "base" (2 in this case), which it happens to be here, 
then we increase out bucket size "base" times, and instead make the bucket of 
*that* size that ends right before our current bucket. So the result will be 
[2000-4000).

This method of getting the next bucket is repeated until we reach timestamp 0. 
Using the above logic, we don't increase the size of the bucket this time, 
because we have a start timestamp of 2000 which becomes 1 when divided by the 
size (2000). So we end up with [0, 2000), and we're done.
The buckets were [4000-5000), [2000-4000) and [0-2000).

What's more important than understanding these rules is of course getting some 
kind of intuition for this. Here's what it boils down to: we want there to be 
"base" equally sized buckets right next to each other before we
*coalesce* them. Every bucket is aligned with its own size (as an analogy, 
compilers typically align 4-byte integers on addresses divisible by 4, same 
concept). So, by extension, the bigger bucket they coalesce into must be 
aligned with *its* size. Not just any "base" adjacent buckets will do, it will 
be those that align with the next size.

The remaining question is when do they coalesce? There will always be at least 
1 and at most "base" buckets of every size. Say "base"=4, then there can be 4 
bucket of some size (by necessity next to each other and aligned on 4 times 
their size). The moment a new bucket of the same size appears, the 4 buckets 
become one and this "fifth" bucket will be alone with its size (and the start 
of a new group of 4 such buckets). (The rule for making the bucket were the 
"now" timestamp lives in, is where new buckets come from).

I wish this was easier to explain in simple terms. I personally find this to 
have very nice properties, in that it gives every bucket a fair amount of time 
to settle before it's time for the next compaction.

Interestingly, I proposed an alternative algorithm in this ticket 
<https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fCASSANDRA-9013&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7ca41a30e8a81144de0cd008d34e90a502%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=QvmtOPUWqYnoFLLY0jx7%2fqgg0tJSleAsdBzo%2bzh22G4%3d>,
 including a patch implementing it. My gut tells me that the mental model that 
you've used here is actually equivalent to that algorithm in the ticket. It's 
just expressed in a very different way. Might be something for me to try to 
prove when I'm really bored :)

Hope this helped! Any particular reason you're investigating this?

/
Bj0rn

On Thu, Mar 17, 2016 at 5:43 PM, Anubhav Kale 
wrote:

> Hello,
>
> I am trying to concretely understand how DTCS makes buckets and I am 
> lookin

RE: Wiki

[Anubhav Kale]

There are others missing as well. For instance, getendpoints (which is quite 
handy).

-Original Message-
From: Dave Brosius [mailto:dbros...@mebigfatguy.com] 
Sent: Sunday, April 3, 2016 8:13 AM
To: dev@cassandra.apache.org
Subject: Re: Wiki

Done

On 04/03/2016 10:56 AM, Pedro Gordo wrote:
> Hi Jonathan
>
> I forgot about that, sorry. It's "PedroGordo".
>
> Best regards
>
> Pedro Gordo
>
> On 3 April 2016 at 15:41, Jonathan Ellis  wrote:
>
>> Hi Pedro,
>>
>> We need your wiki username to add it to the editors list.  Thanks!
>>
>> On Sun, Apr 3, 2016 at 9:15 AM, Pedro Gordo 
>> 
>> wrote:
>>
>>> Hi
>>>
>>> I would like to contribute to C* Wiki if possible, so sending the 
>>> email
>> as
>>> requested on the wiki's front page. The reason for this is that the 
>>> nodetool page 
>>> >> ki.apache.org%2fcassandra%2fNodeTool&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c1d8be3522d2d494617df08d35bd273e5%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=UBQCfJSoDBisqLOYqqk34DE5y8a1YIf3z%2fbl6TRSEqw%3d>
>>>  is missing the "gossipinfo" command. I haven't checked if there's more 
>>> commands missing.
>>>
>>> Let me know if I can help!
>>>
>>> Best regards
>>> Pedro Gordo
>>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder, 
>> https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.d
>> atastax.com&data=01%7c01%7cAnubhav.Kale%40microsoft.com%7c1d8be3522d2
>> d494617df08d35bd273e5%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=Lj
>> hh7VAb8bqQ5aTq6DYIlEXnhAFqyTX7aZeHRqGzCG4%3d
>> @spyced
>>

RE: [Proposal] Mandatory comments

[Anubhav Kale]

As someone not very familiar with core codebase, I just wanted to say that the 
comments on top of StreamSession class are a fantastic example of how comments 
should be. It helped me immensely to dig into node rebuilding code. So, thank 
you very much to the author of those comments.

-Original Message-
From: Sylvain Lebresne [mailto:sylv...@datastax.com] 
Sent: Friday, May 6, 2016 2:22 AM
To: dev@cassandra.apache.org
Subject: Re: [Proposal] Mandatory comments

> When I'm reading code i look for comments to help me understand key
points,
> points that aren't self-evident. If we institute a boilerplate 
> "comment everything" rule then I lose that signpost.

Frankly, I don't love the idea of somewhat limiting comments on purpose so the 
sheer presence of a comment is a "signpost", which kind of imply they are 
fairly rare. Especially because when you write and even review code, many many 
things that appears self-evident because you're deep in the code really aren't 
when you come cold to it (and doubly so when your new to the code). So that 
while commenting should certainly be done intelligently (and there is such 
thing as too much useless comments), you should imo lean toward more rather 
than less commenting in general.

But also, I'm talking here about commenting public methods and classes. I 
certainly agree that for in-code comments you don't want to mechanically 
comment every other line but rather only use comments when there is some 
subtlety.
But I
don't see it mattering that much for classes and methods. You're not gonna read 
the javadoc of a method that you don't use, and if you want to use one, it's 
almost always nice if it has a javadoc (of course, especially if it's a "good" 
one). There is a few cases where that javadoc isn't really useful but no harm 
done either imo. Have you ever think while looking at the javadoc of a library 
"man I wish half of the methods had no javadoc because i'm confused which 
methods are important now".

But anyway, I've worked on code base that had javadoc for every public methods 
and classes and (again provided the comments are good) didn't ever felt that 
this was more confusing than helpful, so I just don't buy that it is. I don't 
deny that there is few public classes and methods that are trivial enough that 
a javadoc don't add anything. And it's not so much that I care about it in 
those case. It's that I think making the rule strict has more pros than cons by 
evacuating any debate over where the line should be for having the javadoc or 
not and focusing our energy elsewhere.

Another of my reasoning is that I suspect one of the main reason our code is 
badly commented is laziness (we all seem to agree that comments are important 
and that our code isn't well commented enough). My hope is that by making some 
minimal level of commenting mandatory, we'll be forced to stop being lazy about 
comments and always be reminded that we should care about them. And I hope that 
this will help improve the overall comment quality.

So hopefully I've explained my reasoning well enough. To sum it up, having a 
javadoc for every public class and method certainly isn't enough (the quality 
of comments is really really important in particular), but I disagree that it 
would hurt, I'm convinced it would be an improvement over our current status 
quo, and more importantly I hope it might be a useful nudge to improve our 
comment situation long term.

But if we can't agree on that, so be it. In that case though, is there a 
concrete alternative proposition to help turn around our current lack of proper 
commenting (I'm specfically interested in fixing our ways here, commenting 
existing code that isn't sufficiently commented is great and we should do it, 
but I feel we should maybe stop committing insufficiently commented code 
first)? Or do we think that having had this conversation will be enough?

--
Sylvain

RemoveNode Behavior Question

[Anubhav Kale]

Hello,

Recently, I started noticing an interesting pattern. When I execute 
"removenode", a subset of the nodes that now own the tokens result it in a CPU 
spike / disk activity, and sometimes SSTables on those nodes shoot up.

After looking through the code, it appears to me that below function forces 
data to be streamed from some of the new nodes to the node from where 
"removenode" is kicked in. Is my understanding correct ?

https://github.com/apache/cassandra/blob/d384e781d6f7c028dbe88cfe9dd3e966e72cd046/src/java/org/apache/cassandra/service/StorageService.java#L2548

Our nodes don't run very hot, but it appears this streaming causes them to have 
issues. If I understand the code correctly, the node that's initiated 
removenode may still not get all the data for moved over ranges. So, what is 
the rationale behind trying to build a "partial replica" ?

Maybe, I am not following this correctly so hoping someone can explain.

Thanks !

RE: RemoveNode Behavior Question

[Anubhav Kale]

But I don't understand how the replica count is getting restored here. The node 
that invoked removenode only owns partial ranges.

-Original Message-
From: Brandon Williams [mailto:dri...@gmail.com] 
Sent: Wednesday, February 22, 2017 10:49 AM
To: dev@cassandra.apache.org
Subject: Re: RemoveNode Behavior Question

Every topology operation tries to respect/restore the RF except for assassinate.

On Wed, Feb 22, 2017 at 12:45 PM, Anubhav Kale < 
anubhav.k...@microsoft.com.invalid> wrote:

> Hello,
>
> Recently, I started noticing an interesting pattern. When I execute 
> "removenode", a subset of the nodes that now own the tokens result it 
> in a CPU spike / disk activity, and sometimes SSTables on those nodes shoot 
> up.
>
> After looking through the code, it appears to me that below function 
> forces data to be streamed from some of the new nodes to the node from 
> where "removenode" is kicked in. Is my understanding correct ?
>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fapache%2Fcassandra%2Fblob%2Fd384e781d6f7c028dbe88cfe9dd3e9&dat
> a=02%7C01%7CAnubhav.Kale%40microsoft.com%7Cf22f2e33447f46c5e82a08d45b5
> 38008%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636233861574178675&
> sdata=NGkgls2RTfWTM7MBJ4MuKdxd7pRZiSRGcWDVUmXwG5Q%3D&reserved=0
> 66e72cd046/src/java/org/apache/cassandra/service/StorageService.java#L
> 2548 <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%
> 2Fgithub.com%2Fapache%2Fcassandra%2Fblob%2Fd384e781d6f7c028dbe88cfe9dd
> 3 e966e72cd046%2Fsrc%2Fjava%2Forg%2Fapache%2Fcassandra%
> 2Fservice%2FStorageService.java%23L2548&data=02%7C01%
> 7CAnubhav.Kale%40microsoft.com%7C173daa48fcaf4ca6498d08d43982318c%
> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636196678720784947&sdata=
> JZ9zWh%2FtJJ%2FbhXXkT41yQhANKaUSBHfP53WraY2vL8M%3D&reserved=0>
>
> Our nodes don't run very hot, but it appears this streaming causes 
> them to have issues. If I understand the code correctly, the node 
> that's initiated removenode may still not get all the data for moved 
> over ranges. So, what is the rationale behind trying to build a "partial 
> replica" ?
>
> Maybe, I am not following this correctly so hoping someone can explain.
>
> Thanks !
>
>

RE: RemoveNode Behavior Question

[Anubhav Kale]

Nevermind. I figured out this is happening on all nodes where the tokens got 
moved. So, explains the big streaming going around in the cluster.

-Original Message-
From: Brandon Williams [mailto:dri...@gmail.com] 
Sent: Wednesday, February 22, 2017 10:53 AM
To: dev@cassandra.apache.org
Subject: Re: RemoveNode Behavior Question

The node that invoked removenode is entirely irrelevant, any node can invoke it.

On Wed, Feb 22, 2017 at 12:51 PM, Anubhav Kale < 
anubhav.k...@microsoft.com.invalid> wrote:

> But I don't understand how the replica count is getting restored here. 
> The node that invoked removenode only owns partial ranges.
>
> -Original Message-
> From: Brandon Williams [mailto:dri...@gmail.com]
> Sent: Wednesday, February 22, 2017 10:49 AM
> To: dev@cassandra.apache.org
> Subject: Re: RemoveNode Behavior Question
>
> Every topology operation tries to respect/restore the RF except for 
> assassinate.
>
> On Wed, Feb 22, 2017 at 12:45 PM, Anubhav Kale < 
> anubhav.k...@microsoft.com.invalid> wrote:
>
> > Hello,
> >
> > Recently, I started noticing an interesting pattern. When I execute 
> > "removenode", a subset of the nodes that now own the tokens result 
> > it in a CPU spike / disk activity, and sometimes SSTables on those 
> > nodes
> shoot up.
> >
> > After looking through the code, it appears to me that below function 
> > forces data to be streamed from some of the new nodes to the node 
> > from where "removenode" is kicked in. Is my understanding correct ?
> >
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> > hu 
> > b.com%2Fapache%2Fcassandra%2Fblob%2Fd384e781d6f7c028dbe88cfe9dd3e9&d
> > at
> > a=02%7C01%7CAnubhav.Kale%40microsoft.com%7Cf22f2e33447f46c5e82a08d45
> > b5 
> > 38008%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63623386157417867
> > 5&
> > sdata=NGkgls2RTfWTM7MBJ4MuKdxd7pRZiSRGcWDVUmXwG5Q%3D&reserved=0
> > 66e72cd046/src/java/org/apache/cassandra/service/StorageService.java
> > #L
> > 2548 
> > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%
> > 2Fgithub.com%2Fapache%2Fcassandra%2Fblob%2Fd384e781d6f7c028dbe88cfe9
> > dd
> > 3 e966e72cd046%2Fsrc%2Fjava%2Forg%2Fapache%2Fcassandra%
> > 2Fservice%2FStorageService.java%23L2548&data=02%7C01%
> > 7CAnubhav.Kale%40microsoft.com%7C173daa48fcaf4ca6498d08d43982318c%
> > 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636196678720784947&sdat
> > a= JZ9zWh%2FtJJ%2FbhXXkT41yQhANKaUSBHfP53WraY2vL8M%3D&reserved=0>
> >
> > Our nodes don't run very hot, but it appears this streaming causes 
> > them to have issues. If I understand the code correctly, the node 
> > that's initiated removenode may still not get all the data for moved 
> > over ranges. So, what is the rationale behind trying to build a 
> > "partial
> replica" ?
> >
> > Maybe, I am not following this correctly so hoping someone can explain.
> >
> > Thanks !
> >
> >
>

Cassandra 2.1.13: Using JOIN_RING=False

[Anubhav Kale]

Hello,

With some inspiration from the Cassandra Summit talk from last year, we are 
trying to setup a cluster with coordinator-only nodes. We setup join_ring=false 
in env.sh, disabled auth in YAML and the nodes are able to start just fine. 
However, we're running into a few problems

1] The nodes marked with join_ring=false continue to store data. Why ?
2] We tried Python driver's whitelistedpolicy. But we notice message like 
below, so we are not able to send queries to all nodes marked as coordinators. 
We also changed the Scala driver to support whitelisting, but see the same 
thing. What are we missing ?
3] Is there any way to concretely tell that only coordinator nodes are getting 
requests from clients ? We don't have OpsCenter.

Thanks !

2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] 
Removing host not found in peers metadata: 
2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host 10.80.10.128 
removed
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host 10.80.10.128
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] 
Removing host not found in peers metadata: 
2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host 10.80.10.127 
removed
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host 10.80.10.127
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] 
Removing host not found in peers metadata: 
2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host 10.80.10.129 
removed
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host 10.80.10.129
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] 
Finished fetching ring info
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] 
Rebuilding token map due to topology changes
2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user functions table not 
found
2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user aggregates table not 
found
2017-05-09 20:45:25,098 [DEBUG] cassandra.cluster: Control connection created
2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection for 
host 10.80.10.125
2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection for 
host 10.80.10.126