Re: Unbalanced cluster

Awesome utility Avi! Thanks for sharing. On Tue, Jul 11, 2017 at 10:57 AM Avi Kivity wrote: > There is now a readme with some examples and a build file. > > On 07/11/2017 11:53 AM, Avi Kivity wrote: > > Yeah, posting a github link carries an implied undertaking to write a > README file and make i

Re: Unbalanced cluster

There is now a readme with some examples and a build file. On 07/11/2017 11:53 AM, Avi Kivity wrote: Yeah, posting a github link carries an implied undertaking to write a README file and make it easily buildable. I'll see what I can do. On 07/11/2017 06:25 AM, Nate McCall wrote: You wou

Re: Unbalanced cluster

Yeah, posting a github link carries an implied undertaking to write a README file and make it easily buildable. I'll see what I can do. On 07/11/2017 06:25 AM, Nate McCall wrote: You wouldnt have a build file laying around for that, would you? On Tue, Jul 11, 2017 at 3:23 PM, Nate McCall

Re: Unbalanced cluster

0 AND gc_grace_seconds = 172000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; The payload column is almost the same size in each record.

Re: Unbalanced cluster

': '60', 'unchecked_tombstone_compaction': >> 'false', 'class': >> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} >> AND compression = {'sstable_compression': >> 'org.apache.cassand

Re: Unbalanced cluster

You wouldnt have a build file laying around for that, would you? On Tue, Jul 11, 2017 at 3:23 PM, Nate McCall wrote: > On Tue, Jul 11, 2017 at 3:20 AM, Avi Kivity wrote: > >> >> >> >> [1] https://github.com/avikivity/shardsim >> > > Avi, that's super handy - thanks for posting. > -- ---

Re: Unbalanced cluster

On Tue, Jul 11, 2017 at 3:20 AM, Avi Kivity wrote: > > > > [1] https://github.com/avikivity/shardsim > Avi, that's super handy - thanks for posting.

Re: Unbalanced cluster

the reason for the default of 256 vnodes is because at that many tokens the random distribution of tokens is enough to balance out each nodes token allocation almost evenly. any less and some nodes will get far more unbalanced, as Avi has shown. In 3.0 there is a new token allocating algorithm howe

Re: Unbalanced cluster

.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 172000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_r

Unbalanced cluster

e.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 172000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND specu

Re: Totally unbalanced cluster

Hi, Regarding the documentation I've already knew: - thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html (From myself, how to handle tombstones) - http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html

Re: Totally unbalanced cluster

Hi, > but it's so easy to add nodes Apache Cassandra has some kind of magic pieces ;-). Sometimes it is dark magic though :p. Yet adding a node is indeed not harder when using NetworkTopologyStrategy, as Jon mentioned above, once the configuration is done once properly. > Number of keys (estim

Re: Totally unbalanced cluster

Adding nodes with NTS is easier, in my opinion. You don’t need to worry about replica placement, if you do it right. > On May 4, 2017, at 7:43 AM, Cogumelos Maravilha > wrote: > > Hi Alain thanks for your kick reply. > > > Regarding SimpleStrategy perhaps you are right but it's so easy to a

Re: Totally unbalanced cluster

Hi Alain thanks for your kick reply. Regarding SimpleStrategy perhaps you are right but it's so easy to add nodes. I'm *not* using vnodes and the default 256. The information that I've posted it a regular nodetool status keyspace. My partition key is a sequencial big int but nodetool cfstatus s

Re: Totally unbalanced cluster

Hi, CREATE KEYSPACE mykeyspace WITH replication = {'class': > 'SimpleStrategy', 'replication_factor': '2'} AND durable_writes = false; The SimpleStrategy is never recommended for production clusters as it does not recognise racks or datacenter, inducing possible availability issues and unpredic

Totally unbalanced cluster

Hi all, I'm using C* 3.10. CREATE KEYSPACE mykeyspace WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '2'} AND durable_writes = false; CREATE TABLE mykeyspace.data ( id bigint PRIMARY KEY, kafka text ) WITH bloom_filter_fp_chance = 0.5 AND caching = {'keys': 'AL

Re: Property file snitch and unbalanced cluster

Hi Robert, What has happened is that you have now two datacenters in your cluster. The way they replicate information will depend on your keyspace settings. Regarding your process I don't think it is safe to do it that way. I'd start off by decommissioning nodes 4 and 5 so that your cluster is ba

Property file snitch and unbalanced cluster

Hi guys, I have a cluster with 5 nodes, cassandra 3.0.5, RF=2, PropertyFileSnitch. Three of the nodes were added initially, DC=PRD. The other two were added more recently. The default for unknown nodes default=DC1:r1 (which is bad in my case, as it would have been nice to have PRD as default) As

Re: Unbalanced cluster with RandomPartitioner

Setting a token outside of the partitioner range sounds like a bug. It's mostly an issue with the RP, but I guess a custom partitioner may also want to validate tokens are within a range. Can you report it to https://issues.apache.org/jira/browse/CASSANDRA Thanks - Aaron Morto

Re: Unbalanced cluster with RandomPartitioner

I thought about our issue again and was thinking, maybe the describeOwnership should take into account, if a token is outside the partitioners maximum token range? To recap our problem: we had tokens, that were apart by 12.5% of the token range 2**127, however, we had an offset on each token, w

Re: Unbalanced cluster with RandomPartitioner

Thanks for all the responses! I found our problem: Using the Random Partitioner, the key range is from 0..2**127.When we added nodes, we generated the keys and out of convenience, we added an offset to the tokens because the move was easier like that. However, we did not execute the modulo 2**1

Re: Unbalanced cluster with RandomPartitioner

On 19.01.2012, at 20:15, Narendra Sharma wrote: > I believe you need to move the nodes on the ring. What was the load on the > nodes before you added 5 new nodes? Its just that you are getting data in > certain token range more than others. With three nodes, it was also imbalanced. What I don't

Re: Unbalanced cluster with RandomPartitioner

I believe you need to move the nodes on the ring. What was the load on the nodes before you added 5 new nodes? Its just that you are getting data in certain token range more than others. -Naren On Thu, Jan 19, 2012 at 3:22 AM, Marcel Steinbach wrote: > On 18.01.2012, at 02:19, Maki Watanabe wro

Re: Unbalanced cluster with RandomPartitioner

Load reported from node tool ring is the live load, which means SSTables that the server has open and will read from during a request. This will include tombstones, expired and over written data. nodetool ctstats also includes "dead" load, which is sstables that are in use but still on disk.

Re: Unbalanced cluster with RandomPartitioner

2012/1/19 aaron morton : > If you have performed any token moves the data will not be deleted until you > run nodetool cleanup. We did that after adding nodes to the cluster. And then, the cluster wasn't balanced either. Also, does the "Load" really account for "dead" data, or is it just live data?

Re: Unbalanced cluster with RandomPartitioner

On 18.01.2012, at 02:19, Maki Watanabe wrote: > Are there any significant difference of number of sstables on each nodes? No, no significant difference there. Actually, node 8 is among those with more sstables but with the least load (20GB) On 17.01.2012, at 20:14, Jeremiah Jordan wrote: > Are yo

Re: Unbalanced cluster with RandomPartitioner

If you have performed any token moves the data will not be deleted until you run nodetool cleanup. To get a baseline I would run nodetool compact to do major compaction and purge any tomb stones as others have said. Cheers - Aaron Morton Freelance Developer @aaronmorton http:

Re: Unbalanced cluster with RandomPartitioner

Are there any significant difference of number of sstables on each nodes? 2012/1/18 Marcel Steinbach : > We are running regular repairs, so I don't think that's the problem. > And the data dir sizes match approx. the load from the nodetool. > Thanks for the advise, though. > > Our keys are digits

Re: Unbalanced cluster with RandomPartitioner

Are you deleting data or using TTL's? Expired/deleted data won't go away until the sstable holding it is compacted. So if compaction has happened on some nodes, but not on others, you will see this. The disparity is pretty big 400Gb to 20GB, so this probably isn't the issue, but with our dat

Re: Unbalanced cluster with RandomPartitioner

We are running regular repairs, so I don't think that's the problem. And the data dir sizes match approx. the load from the nodetool. Thanks for the advise, though. Our keys are digits only, and all contain a few zeros at the same offsets. I'm not that familiar with the md5 algorithm, but I doub

Re: Unbalanced cluster with RandomPartitioner

Have you tried running repair first on each node? Also, verify using df -h on the data dirs On Tue, Jan 17, 2012 at 7:34 AM, Marcel Steinbach wrote: > Hi, > > we're using RP and have each node assigned the same amount of the token > space. The cluster looks like that: > > Address         Status

Unbalanced cluster with RandomPartitioner

Hi, we're using RP and have each node assigned the same amount of the token space. The cluster looks like that: Address Status State LoadOwnsToken 205648943402372032879374446

Re: how to solve one node is in heavy load in unbalanced cluster

gt;>>>> nodetool -h node3 ring >>>>> Address Status State LoadOwnsToken >>>>> >>>>> >>>>> 84944475733633104818662955375549269696 >>>>> node1 Up Normal 13.18 GB81.09% >>>>> 52773518586096316348543097376923124102 >>>>> node2 Up Normal 22.85 GB10.48% >>>>> 70597222385644499881390884416714081360 >>>>> node3 Up Leaving 25.44 GB8.43% >>>>> 84944475733633104818662955375549269696 >>>>> >>>>> the log didn't show any error message neither anything abnormal. is >>>>> there something wrong? >>>>> >>>>> >>>>> I used to have RF=2, and changed it to RF=3 using cassandra-cli. >>>>> >>>>> >>>>> On Mon, Aug 1, 2011 at 10:22 AM, Yan Chunlu wrote: >>>>> >>>>>> thanks a lot! I will try the "move". >>>>>> >>>>>> >>>>>> On Mon, Aug 1, 2011 at 7:07 AM, mcasandra wrote: >>>>>> >>>>>>> >>>>>>> springrider wrote: >>>>>>> > >>>>>>> > is that okay to do nodetool move before a completely repair? >>>>>>> > >>>>>>> > using this equation? >>>>>>> > def tokens(nodes): >>>>>>> > >>>>>>> >- for x in xrange(nodes): >>>>>>> > - print 2 ** 127 / nodes * x >>>>>>> > >>>>>>> >>>>>>> Yes use that logic to get the tokens. I think it's safe to run move >>>>>>> first >>>>>>> and reair later. You are moving some nodes data as is so it's no >>>>>>> worse than >>>>>>> what you have right now. >>>>>>> >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html >>>>>>> Sent from the cassandra-u...@incubator.apache.org mailing list >>>>>>> archive at Nabble.com. >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > >

Re: how to solve one node is in heavy load in unbalanced cluster

e before a completely repair? > > > > using this equation? > > def tokens(nodes): > > > >- for x in xrange(nodes): > > - print 2 ** 127 / nodes * x > > > > Yes use that logic to get the tokens. I think it's safe to run move first > and reair later. You are moving some nodes data as is so it's no worse than > what you have right now. > > -- > View this message in context: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html > Sent from the cassandra-u...@incubator.apache.org mailing list archive at > Nabble.com. > > > > > >

Re: how to solve one node is in heavy load in unbalanced cluster

96316348543097376923124102 >>>> node2 Up Normal 22.85 GB10.48% >>>> 70597222385644499881390884416714081360 >>>> node3 Up Leaving 25.44 GB8.43% >>>> 84944475733633104818662955375549269696 >>>> >>>> the log didn't show any error message neither anything abnormal. is >>>> there something wrong? >>>> >>>> >>>> I used to have RF=2, and changed it to RF=3 using cassandra-cli. >>>> >>>> >>>> On Mon, Aug 1, 2011 at 10:22 AM, Yan Chunlu wrote: >>>> >>>>> thanks a lot! I will try the "move". >>>>> >>>>> >>>>> On Mon, Aug 1, 2011 at 7:07 AM, mcasandra wrote: >>>>> >>>>>> >>>>>> springrider wrote: >>>>>> > >>>>>> > is that okay to do nodetool move before a completely repair? >>>>>> > >>>>>> > using this equation? >>>>>> > def tokens(nodes): >>>>>> > >>>>>> >- for x in xrange(nodes): >>>>>> > - print 2 ** 127 / nodes * x >>>>>> > >>>>>> >>>>>> Yes use that logic to get the tokens. I think it's safe to run move >>>>>> first >>>>>> and reair later. You are moving some nodes data as is so it's no worse >>>>>> than >>>>>> what you have right now. >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html >>>>>> Sent from the cassandra-u...@incubator.apache.org mailing list >>>>>> archive at Nabble.com. >>>>>> >>>>> >>>>> >>>> >>> >> >

Re: how to solve one node is in heavy load in unbalanced cluster

5549269696 >>> >>> the log didn't show any error message neither anything abnormal. is >>> there something wrong? >>> >>> >>> I used to have RF=2, and changed it to RF=3 using cassandra-cli. >>> >>> >>> On Mon, Aug 1, 2011 at 10:22 AM, Yan Chunlu wrote: >>> >>>> thanks a lot! I will try the "move". >>>> >>>> >>>> On Mon, Aug 1, 2011 at 7:07 AM, mcasandra wrote: >>>> >>>>> >>>>> springrider wrote: >>>>> > >>>>> > is that okay to do nodetool move before a completely repair? >>>>> > >>>>> > using this equation? >>>>> > def tokens(nodes): >>>>> > >>>>> >- for x in xrange(nodes): >>>>> > - print 2 ** 127 / nodes * x >>>>> > >>>>> >>>>> Yes use that logic to get the tokens. I think it's safe to run move >>>>> first >>>>> and reair later. You are moving some nodes data as is so it's no worse >>>>> than >>>>> what you have right now. >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html >>>>> Sent from the cassandra-u...@incubator.apache.org mailing list archive >>>>> at Nabble.com. >>>>> >>>> >>>> >>> >> >

Re: how to solve one node is in heavy load in unbalanced cluster

n, Aug 1, 2011 at 10:22 AM, Yan Chunlu wrote: >> >>> thanks a lot! I will try the "move". >>> >>> >>> On Mon, Aug 1, 2011 at 7:07 AM, mcasandra wrote: >>> >>>> >>>> springrider wrote: >>>> > >>>> > is that okay to do nodetool move before a completely repair? >>>> > >>>> > using this equation? >>>> > def tokens(nodes): >>>> > >>>> >- for x in xrange(nodes): >>>> > - print 2 ** 127 / nodes * x >>>> > >>>> >>>> Yes use that logic to get the tokens. I think it's safe to run move >>>> first >>>> and reair later. You are moving some nodes data as is so it's no worse >>>> than >>>> what you have right now. >>>> >>>> -- >>>> View this message in context: >>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html >>>> Sent from the cassandra-u...@incubator.apache.org mailing list archive >>>> at Nabble.com. >>>> >>> >>> >> >

Re: how to solve one node is in heavy load in unbalanced cluster

gt; > >>> > is that okay to do nodetool move before a completely repair? >>> > >>> > using this equation? >>> > def tokens(nodes): >>> > >>> > - for x in xrange(nodes): >>> > - print 2 ** 127 /

Re: how to solve one node is in heavy load in unbalanced cluster

I think it's safe to run move first >> and reair later. You are moving some nodes data as is so it's no worse >> than >> what you have right now. >> >> -- >> View this message in context: >> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html >> Sent from the cassandra-u...@incubator.apache.org mailing list archive at >> Nabble.com. >> > >

Re: how to solve one node is in heavy load in unbalanced cluster

t; > -- > View this message in context: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6639317.html > Sent from the cassandra-u...@incubator.apache.org mailing list archive at > Nabble.com. >

Re: how to solve one node is in heavy load in unbalanced cluster

; nodetool move you will need to determine tokens that each node will be >> responsible for. Then use that token to perform move. >> >> -- >> View this message in context: >> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6638649.html >> Sent from the cassandra-u...@incubator.apache.org mailing list archive at >> Nabble.com. >> > > >

Re: how to solve one node is in heavy load in unbalanced cluster

o run move first and reair later. You are moving some nodes data as is so it's no worse than what you have right now. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6

Re: how to solve one node is in heavy load in unbalanced cluster

p://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6638649.html > Sent from the cassandra-u...@incubator.apache.org mailing list archive at > Nabble.com. >

Re: how to solve one node is in heavy load in unbalanced cluster

you > run > nodetool move you will need to determine tokens that each node will be > responsible for. Then use that token to perform move. > > -- > View this message in context: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/how-to-solve-one-node-is-in-heavy-lo

Re: how to solve one node is in heavy load in unbalanced cluster

.nabble.com/how-to-solve-one-node-is-in-heavy-load-in-unbalanced-cluster-tp6630827p6638649.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: how to solve one node is in heavy load in unbalanced cluster

any help? thanks! On Fri, Jul 29, 2011 at 12:05 PM, Yan Chunlu wrote: > and by the way, my RF=3 and the other two nodes have much more capacity, > why does they always routed the request to node3? > > coud I do a rebalance now? before node repair? > > > On Fri, Jul 29, 2011 at 12:01 PM, Yan Chun

Re: how to solve one node is in heavy load in unbalanced cluster

and by the way, my RF=3 and the other two nodes have much more capacity, why does they always routed the request to node3? coud I do a rebalance now? before node repair? On Fri, Jul 29, 2011 at 12:01 PM, Yan Chunlu wrote: > add new nodes seems added more pressure to the cluster? how about you

Re: how to solve one node is in heavy load in unbalanced cluster

add new nodes seems added more pressure to the cluster? how about your data size? On Fri, Jul 29, 2011 at 4:16 AM, Frank Duan wrote: > "Dropped read message" might be an indicator of capacity issue. We > experienced the similar issue with 0.7.6. > > We ended up adding two extra nodes and physi

Re: how to solve one node is in heavy load in unbalanced cluster

"Dropped read message" might be an indicator of capacity issue. We experienced the similar issue with 0.7.6. We ended up adding two extra nodes and physically rebooted the offending node(s). The entire cluster then calmed down. On Thu, Jul 28, 2011 at 2:24 PM, Yan Chunlu wrote: > I have three

how to solve one node is in heavy load in unbalanced cluster

I have three nodes and RF=3.here is the current ring: Address Status State Load Owns Token 84944475733633104818662955375549269696 node1 Up Normal 15.32 GB 81.09% 52773518586096316348543097376923124102 node2 Up Normal 22.51 GB 10.48% 70597222385644499881390884416714081360 node3 Up Normal 56.1 GB

Re: fixing unbalanced cluster !?

Thanks Ben. That's what I was afraid I had to do. I can see how it's a lot easier if you simply double the cluster when adding capacity. Jon On Jun 9, 2011, at 4:44 PM, Benjamin Coverston wrote: > Because you were able to successfully run repair you can follow up with a > nodetool cleanup

Re: fixing unbalanced cluster !?

Because you were able to successfully run repair you can follow up with a nodetool cleanup which will git rid of some of the extraneous data on that (bigger) node. You're also assured after you run repair that entropy beteen the nodes is minimal. Assuming you're using the random ordered partit

fixing unbalanced cluster !?

I got myself into a situation where one node (10.47.108.100) has a lot more data than the other nodes. In fact, the 1 TB disk on this node is almost full. I added 3 new nodes and let cassandra automatically calculate new tokens by taking the highest loaded nodes. Unfortunately there is still