We also run into the same problem when trying to load data from a 8 node
cluster (C* 1.2.1, Vnodes and same rack) into a 9 node cluster (C* 1.2.11,
Vnodes and different racks) using sstableloader.
We observed that a Key that clearly belonged to 3 nodes (using nodetool
getendpoints) has not been
As an update to this thread, we conducted several tests with
Cassandra-1.2.9, varying parameters such as partitioner
(Murmur3Partitioner/RandomParttioner), using NetworkToplogyStrategy (with
Ec2Snitch) / SimpleStrategy (with SimpleSnitch) across 2 Availability zones
and 1 AZ. We also tested the con
On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar <
suruchi.deod...@generalsentiment.com> wrote:
> Using the nodes in the same availability zone(us-east-1b), we still get a
> highly imbalanced cluster. The nodetool status and ring output is attached.
> Even after running repairs, the cluster does n
Did you start out your cluster after wiping all the sstables and commit
logs?
On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar <
suruchi.deod...@generalsentiment.com> wrote:
> We have been trying to resolve this issue to find a stable configuration
> that can give us a balanced cluster with equal
On Fri, Sep 20, 2013 at 9:24 AM, Jayadev Jayaraman wrote:
> As a follow-up, is operating a Cassandra cluster with machines on multiple
> racks and vnodes bound to cause load imbalance ? Shouldn't token-ranges
> assigned to individual machines via their vnodes be approximately balanced
> ? We're ot
As a follow-up, is operating a Cassandra cluster with machines on multiple
racks and vnodes bound to cause load imbalance ? Shouldn't token-ranges
assigned to individual machines via their vnodes be approximately balanced
? We're otherwise unable to explain why this imbalance occurs. ( it
shouldn't
Like I said in my previous reply that I am not sure if that is the
problem and that's why I thought it would be a good test to do your test
with cluster in one RACK only.
I'll take a look at your ring output today. Did you also post cfstats
output?
On Fri, Sep 20, 2013 at 9:24 AM, Jayadev Jayaram
Other thing I noticed is that you are using mutiple RACKS and that might be
contributing factor to it. However, I am not sure.
Can you paste the output of nodetool cfstats and ring? Is it possible to
run the same test but keeping all the nodes in one rack?
I think you should open a JIRA if you ar
We ran nodetool repair on all nodes for all Keyspaces / CFs, restarted
cassandra and this is what we get for nodetool status :
bin/nodetool -h localhost status
Datacenter: us-east
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns
Can you run nodetool repair on all the nodes first and look at the keys?
On Thu, Sep 19, 2013 at 1:22 PM, Suruchi Deodhar <
suruchi.deod...@generalsentiment.com> wrote:
> Yes, the key distribution does vary across the nodes. For example, on the
> node with the highest data, Number of Keys (estima
On 19 September 2013 20:36, Suruchi Deodhar <
suruchi.deod...@generalsentiment.com> wrote:
> Thanks for your replies. I wiped out my data from the cluster and also
> cleared the commitlog before restarting it with num_tokens=256. I then
> uploaded data using sstableloader.
>
> However, I am still
Can you check cfstats to see number of keys per node?
On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar <
suruchi.deod...@generalsentiment.com> wrote:
> Thanks for your replies. I wiped out my data from the cluster and also
> cleared the commitlog before restarting it with num_tokens=256. I then
Yes, the key distribution does vary across the nodes. For example, on the
node with the highest data, Number of Keys (estimate) is 6527744 for a
particular column family, whereas for the same column family on the node
with least data, Number of Keys (estimate) = 3840.
Is there a way to control thi
Thanks for your replies. I wiped out my data from the cluster and also
cleared the commitlog before restarting it with num_tokens=256. I then
uploaded data using sstableloader.
However, I am still not able to see a uniform distribution of data across
nodes of the clusters.
The output of the bin/n
The only thing you need to guarantee is that Cassandra doesn't start with
num_tokens=1 (the default in 1.2.x) or, if it does, that you wipe all the
data before starting it with higher num_tokens.
On 19 September 2013 19:07, Robert Coli wrote:
> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar
On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar <
suruchi.deod...@generalsentiment.com> wrote:
> Do you suggest I should try with some other installation mechanism? Are
> there any known problems with the tar installation of cassandra 1.2.9 that
> I should be aware of?
>
I was asking in the con
Hi Rob,
Do you suggest I should try with some other installation mechanism? Are
there any known problems with the tar installation of cassandra 1.2.9 that
I should be aware of? Please do let me know.
Thanks,
Suruchi
On Thu, Sep 19, 2013 at 1:04 PM, Suruchi Deodhar <
suruchi.deod...@generalsentime
Hi Robert,
I downloaded apache-cassandra-1.2.9.tar.gz from
http://cassandra.apache.org/download/ (
http://apache.mirrors.tds.net/cassandra/1.2.9/apache-cassandra-1.2.9-bin.tar.gz)
and installed it on the individual nodes of the cassandra cluster.
Thanks,
Suruchi
On Thu, Sep 19, 2013 at 12:35 PM,
On Thu, Sep 19, 2013 at 7:03 AM, Richard Low wrote:
> I think what has happened is that Cassandra was started with num_tokens =
> 1, then shutdown and num_tokens set to 256. When this happens, the first
> time Cassandra chooses a single random token. Then when restarted it
> splits the token in
I think what has happened is that Cassandra was started with num_tokens =
1, then shutdown and num_tokens set to 256. When this happens, the first
time Cassandra chooses a single random token. Then when restarted it
splits the token into 256 adjacent ranges.
You can see something like this has h
Hi Richard,
This is a brand new cluster which started with num_tokens =256 on first boot
and chose random tokens. The attached ring status is after data is loaded into
the cluster for the first time using sdtableloader and remains that way even
after Cassandra is restarted.
Thanks,
Suruchi
On
On 19 September 2013 02:06, Jayadev Jayaraman wrote:
We use vnodes with num_tokens = 256 ( 256 tokens per node ) . After loading
> some data with sstableloader , we find that the cluster is heavily
> imbalanced :
>
How did you select the tokens? Is this a brand new cluster which started
on firs
22 matches
Mail list logo
