> Just wanted to follow up on this.
>
> We were never able to achieve throughput scaling in the cloud. We were able
> to verify that many of our cluster nodes and test servers were collocated on
> the same physical hardware (thanks Stu for the tip on the Rackspace REST
> API), and that performa
Just wanted to follow up on this.
We were never able to achieve throughput scaling in the cloud. We were able to
verify that many of our cluster nodes and test servers were collocated on the
same physical hardware (thanks Stu for the tip on the Rackspace REST API), and
that performance on coll
> (I'm hoping to have time to run my test on EC2 tonight; will see.)
Well, I needed three c1.xlarge EC2 instances running py_stress to even
saturate more than one core on the c1.xlarge instance running a single
cassandra node (at roughly 21k reqs/second)... Depending on how
reliable vmstat/top is
On Tue, Jul 20, 2010 at 6:20 AM, Juho Mäkinen wrote:
> I managed to run a few benchmarks.
>
> Servers r/s
> 1 64.5k
> 2 59.5k
>
> The configuration:
> Client: Machine with four Quad Core Intel Xeon CPU E5520 @ 2.27Ghz
> cpus (total 16 cores), 4530 bogomips per core. 12 GB ECC c
> But what's then the point with adding nodes into the ring? Disk speed!
Well, it may also be cheaper to service an RPC request than service a
full read or write, even in terms of CPU.
But: Even taking into account that requests are distributed randomly,
the cluster should still scale. You will a
I managed to run a few benchmarks.
Servers r/s
164.5k
259.5k
The configuration:
Client: Machine with four Quad Core Intel Xeon CPU E5520 @ 2.27Ghz
cpus (total 16 cores), 4530 bogomips per core. 12 GB ECC corrected
memory. Supermicro mainboard (not sure about exact type).
I've put up a bunch of steps to get Cassandra installed on an EC2 instance:
http://wiki.apache.org/cassandra/CloudConfig
Look at the "step-by-step guide".
I haven't AMI-ed the result, since the steps are fairly quick and it would
be just one more thing to update with a new release of Cassandra...
> Now keep adding clients until it stops making the numbers go up...
Neither adding additional readers nor additional cluster nodes showed
performance gains. The numbers, they do not move.
--
David Schoonover
On Jul 19, 2010, at 5:18 PM, Jonathan Ellis wrote:
> Now keep adding clients until i
> CPU was approximately equal across the cluster; it was around 50%.
>
> stress.py generates keys randomly or using a gaussian distribution, both
> methods showed the same results.
>
> Finally, we're using a random partitioner, so Cassandra will hash the keys
> using md5 to map it to a position o
Now keep adding clients until it stops making the numbers go up...
On Mon, Jul 19, 2010 at 2:51 PM, David Schoonover
wrote:
> Sorry, mixed signals in my response. I was partially replying to suggestions
> that we were limited by the box's NIC or DC's bandwidth (which is gigabit, no
> dice there
> Did you see about equal CPU usage on the cassandra nodes during the
> test? Is it possible that most or all of the keys generated by
> stress.py simply fall on a single node?
CPU was approximately equal across the cluster; it was around 50%.
stress.py generates keys randomly or using a gaussian
Thanks a ton, Juho.
The command was:
./stress.py -o read -t 50 -d $NODELIST -n 7500 -k -i 2
I made a few minor modifications to stress.py to count errors instead of
logging them, and avoid the pointless try-catch on missing keys. (There are
also unrelated edits to restart long run
I'm about to extend my two node cluster with four dedicated nodes and
removing one of the old nodes, leaving a five node cluster. The
cluster is in production, but I can spare it to do some stress testing
in the meantime as I'm also interested about my cluster performance. I
can't dedicate the clus
The following is completely irrelevant if you are indeed using the
default storage-conf.xml as you said. However since I wrote it and it
remains relevant for anyone testing with the order preserving
partitioner, I might aswell post it rather than discard it...
Begin probably irrelevant post:
Anot
I'll just add that CPU usage hovered around 50% during these tests.
On Jul 19, 2010, at 3:51 PM, David Schoonover wrote:
> Sorry, mixed signals in my response. I was partially replying to suggestions
> that we were limited by the box's NIC or DC's bandwidth (which is gigabit, no
> dice there).
Sorry, mixed signals in my response. I was partially replying to suggestions
that we were limited by the box's NIC or DC's bandwidth (which is gigabit, no
dice there). I also ran the tests with -t50 on multiple tester machines in the
cloud with no change in performance; I've now rerun those test
> stress.py uses multiprocessing if it is present, circumventing the GIL; we
> ran the tests with python 2.6.5.
Ah, sorry about that. I was mis-remembering because I had to use
threading with pystress because multiprocessing was broken/unavailabie
(can't remember which) on FreeBSD.
I agree with
Usually a fixed bottleneck results from a limited resource -- you've
eliminated disk from the test and you don't mention that CPU is a serious
issue, or memory for that matter.
So for me that leaves network i/o and switch capacity. Is it possible that
your test is saturating your local network ca
On Mon, Jul 19, 2010 at 11:02 AM, David Schoonover
wrote:
>> Multiple client processes, or multiple client machines?
>
>
> I ran it with both one and two client machines making requests, and ensured
> the sum of the request threads across the clients was 50. That was on the
> cloud. I am re-runn
I'm reading what this thread and I am a little lost, what should the
expected behavioral be?
Should it maintain 53K regardless of nodes?
nodes reads/sec
1 53,000
2 37,000
4 37,000
I ran this test previously on the cloud, with similar results:
nodes reads/sec
1 24,000
additional gains: above that point, you should use
more machines, each running K processes.
-Original Message-
From: "David Schoonover"
Sent: Monday, July 19, 2010 1:02pm
To: user@cassandra.apache.org
Subject: Re: Cassandra benchmarking on Rackspace Cloud
> Multiple client
> Multiple client processes, or multiple client machines?
I ran it with both one and two client machines making requests, and ensured the
sum of the request threads across the clients was 50. That was on the cloud. I
am re-running the multi-host test against the 4-node cluster on dedicated
har
stress.py uses multiprocessing if it is present, circumventing the GIL; we ran
the tests with python 2.6.5.
David Schoonover
On Jul 19, 2010, at 1:51 PM, Peter Schuller wrote:
>>> One with 50 threads; it is remote from the cluster but within the same
>>> DC in both cases. I also run the test wi
>> One with 50 threads; it is remote from the cluster but within the same
>> DC in both cases. I also run the test with multiple clients and saw
>> similar results when summing the reqs/sec.
>
> Multiple client processes, or multiple client machines?
In particular, note that the way CPython works,
On Mon, Jul 19, 2010 at 12:30 PM, David Schoonover
wrote:
>> How many physical client machines are running stress.py?
>
> One with 50 threads; it is remote from the cluster but within the same
> DC in both cases. I also run the test with multiple clients and saw
> similar results when summing the
u Hood wrote:
> How many physical client machines are running stress.py?
>
> -Original Message-
> From: "David Schoonover"
> Sent: Monday, July 19, 2010 12:11pm
> To: user@cassandra.apache.org
> Subject: Re: Cassandra benchmarking on Rackspace Cloud
>
>
> Another thing: Is the py_stress traffic definitely non-determinstic
> such that each client will generate a definitely unique series of
> requests?
The tests were run both with --random and --std 0.1; in both cases, the
key-sequence is non-deterministic.
Cheers,
Dave
On Jul 19, 2010, at 1
ote:
> How many physical client machines are running stress.py?
>
> -Original Message-
> From: "David Schoonover"
> Sent: Monday, July 19, 2010 12:11pm
> To: user@cassandra.apache.org
> Subject: Re: Cassandra benchmarking on Rackspace Cloud
>
> Hello all,
Another thing: Is the py_stress traffic definitely non-determinstic
such that each client will generate a definitely unique series of
requests? If all clients are deterministically requesting the same
sequence of keys, it would otherwise be plausible that they end up in
effective lock-step, if the
This may be too much work... but you might consider building an Amazon EC2
AMI of your nodes. This would let others quickly boot up your nodes and run
the stress test against it.
I know you mentioned that you're using Rackspace Cloud. I'm not super
familiar with the internals of RSCloud, but per
> I ran this test previously on the cloud, with similar results:
>
> nodes reads/sec
> 1 24,000
> 2 21,000
> 3 21,000
> 4 21,000
> 5 21,000
> 6 21,000
>
> In fact, I ran it twice out of disbelief (on different nodes the second time)
> to essentially identical
How many physical client machines are running stress.py?
-Original Message-
From: "David Schoonover"
Sent: Monday, July 19, 2010 12:11pm
To: user@cassandra.apache.org
Subject: Re: Cassandra benchmarking on Rackspace Cloud
Hello all, I'm Oren's partner in crime on
ot;Oren Benjamin"
> Sent: Monday, July 19, 2010 10:30am
> To: "user@cassandra.apache.org"
> Subject: Re: Cassandra benchmarking on Rackspace Cloud
>
> Certainly I'm using multiple cloud servers for the multiple client tests.
> Whether or not they are res
Oren Benjamin"
Sent: Monday, July 19, 2010 10:30am
To: "user@cassandra.apache.org"
Subject: Re: Cassandra benchmarking on Rackspace Cloud
Certainly I'm using multiple cloud servers for the multiple client tests.
Whether or not they are resident on the same physical machine,
Yes, as the size of the data on disk increases and the OS cannot avoid disk
seeks the read performance degrades. You can see this in the results from the
original post where the number of keys in the test goes from 10M to 100M the
reads drop from 4,600/s to 200/s. 10M keys in the stress.py tes
Certainly I'm using multiple cloud servers for the multiple client tests.
Whether or not they are resident on the same physical machine, I just don't
know.
-- Oren
On Jul 18, 2010, at 11:35 PM, Brandon Williams wrote:
On Sun, Jul 18, 2010 at 8:45 PM, Oren Benjamin
mailto:o...@clearspring.
On Sun, Jul 18, 2010 at 8:45 PM, Oren Benjamin wrote:
> Thanks for the info. Very helpful in validating what I've been seeing. As
> for the scaling limit...
>
> >> The above was single node testing. I'd expect to be able to add nodes
> and scale throughput. Unfortunately, I seem to be running
Thanks for the info. Very helpful in validating what I've been seeing. As for
the scaling limit...
>> The above was single node testing. I'd expect to be able to add nodes and
>> scale throughput. Unfortunately, I seem to be running into a cap of 21,000
>> reads/s regardless of the number o
On Sat, Jul 17, 2010 at 1:31 PM, Brandon Williams wrote:
> Most of the data was hot in either the row/key cache, or in the OS file
> cache. The point was to benchmark Cassandra, not how fast the disk can
> seek.
IIRC we were running w/ the defaults of 200k key cache and no row cache.
--
Jonath
On Sat, Jul 17, 2010 at 12:02 PM, Schubert Zhang wrote:
> Hi Jonathan,
> The 7k reads/s is very high, could you please make more explain about your
> benchmark?
>
> 7000 reads/s makes average latency of each read operation only talks
> 0.143ms. Consider 2 disks in the benchmark, it may be 0.286ms
I fact, in my cassandra-0.6.2, I can only get about 40~50 reads/s with
disabled Key/Row cache.
On Sun, Jul 18, 2010 at 1:02 AM, Schubert Zhang wrote:
> Hi Jonathan,
> The 7k reads/s is very high, could you please make more explain about your
> benchmark?
>
> 7000 reads/s makes average latency of
Hi Jonathan,
The 7k reads/s is very high, could you please make more explain about your
benchmark?
7000 reads/s makes average latency of each read operation only talks
0.143ms. Consider 2 disks in the benchmark, it may be 0.286ms.
But in most random read applications on very large dataset, OS cac
On Fri, Jul 16, 2010 at 6:06 PM, Oren Benjamin wrote:
> The first goal was to reproduce the test described on spyced here:
> http://spyced.blogspot.com/2010/01/cassandra-05.html
>
> Using Cassandra 0.6.3, a 4GB/160GB cloud server
> (http://www.rackspacecloud.com/cloud_hosting_products/servers/pr
I think your read throughput is very high, and it may be unauthentic.
For random read, the disk seek will always be the bottleneck (100% utils)
There will be about 3 random disk-seeks for a random read, and aout 10ms for
one seek. So, there will be 30ms for a random read.
If you have only one dis
I've been doing quite a bit of benchmarking of Cassandra in the cloud using
stress.py I'm working on a comprehensive spreadsheet of results with a
template that others can add to, but for now I thought I'd post some of the
basic results here to get some feedback from others.
The first goal was
45 matches
Mail list logo
