Re: Cassandra Doesn't Get Linear Performance Increment in Stress Test on Amazon EC2

2014-12-08 Thread Eric Stevens
> Do you mean that I should clear my table after each run? Indeed, I can see several times of compaction during my test, but could only a few times compaction affect the performance that much? It certainly affects performance. Read performance suffers first, then write performance suffers eventua

Re: Cassandra Doesn't Get Linear Performance Increment in Stress Test on Amazon EC2

2014-12-08 Thread Chris Lohfink
So I would -expect- an increase of ~20k qps per node with m3.xlarge so there may be something up with your client (I am not a c++ person however but hopefully someone on list will take notice). Latency does not decreases linearly as you add nodes. What you are likely seeing with latency since so

Re: Cassandra Doesn't Get Linear Performance Increment in Stress Test on Amazon EC2

2014-12-08 Thread 孔嘉林
Thanks Chris. I run a *client on a separate* AWS *instance from* the Cassandra cluster servers. At the client side, I create 40 or 50 threads for sending requests to each Cassandra node. I create one thrift client for each of the threads. And at the beginning, all the created thrift clients connect

Re: Cassandra Doesn't Get Linear Performance Increment in Stress Test on Amazon EC2

2014-12-07 Thread Chris Lohfink
I think your client could use improvements. How many threads do you have running in your test? With a thrift call like that you only can do one request at a time per connection. For example, assuming C* takes 0ms, a 10ms network latency/driver overhead will mean 20ms RTT and a max throughput of

Re: Cassandra Doesn't Get Linear Performance Increment in Stress Test on Amazon EC2

2014-12-07 Thread 孔嘉林
Hi Eric, Thank you very much for your reply! Do you mean that I should clear my table after each run? Indeed, I can see several times of compaction during my test, but could only a few times compaction affect the performance that much? Also, I can see from the OpsCenter some ParNew GC happen but no

Re: Cassandra Doesn't Get Linear Performance Increment in Stress Test on Amazon EC2

2014-12-07 Thread Eric Stevens
I'm sorry, I meant to say "6 nodes rf=3". Also look at this performance over sustained periods of times, not burst writing. Run your test for several hours and watch memory and especially compaction stats. See if you can walk in what data volume you can write while keeping outstanding compaction

Re: Cassandra Doesn't Get Linear Performance Increment in Stress Test on Amazon EC2

2014-12-07 Thread Eric Stevens
Hi Joy, Are you resetting your data after each test run? I wonder if your tests are actually causing you to fall behind on data grooming tasks such as compaction, and so performance suffers for your later tests. There are *so many* factors which can affect performance, without reviewing test met

Cassandra Doesn't Get Linear Performance Increment in Stress Test on Amazon EC2

2014-12-06 Thread kong
Hi, I am doing stress test on Datastax Cassandra Community 2.1.2, not using the provided stress test tool, but use my own stress-test client code instead(I write some C++ stress test code). My Cassandra cluster is deployed on Amazon EC2, using the provided Datastax Community AMI( HVM instances ) i