Re: Increased latency after setting row_cache_size_in_mb

2018-02-06 Thread mohsin k
@Rahul, I am using cassandra-stress tool. On Tue, Feb 6, 2018 at 7:37 PM, Rahul Singh wrote: > Could be the cause. I would run 2 and then 4 concurrent clients to see how > they behave. What’s your client written in? How are you managing your > connection? > > -- > Rahul Singh > rahul.si...@anant

Re: Increased latency after setting row_cache_size_in_mb

2018-02-06 Thread Rahul Singh
Could be the cause. I would run 2 and then 4 concurrent clients to see how they behave. What’s your client written in? How are you managing your connection? -- Rahul Singh rahul.si...@anant.us Anant Corporation On Feb 6, 2018, 8:50 AM -0500, mohsin k , wrote: > Thanks, Jeff, will definitely che

Re: Increased latency after setting row_cache_size_in_mb

2018-02-06 Thread mohsin k
Thanks, Jeff, will definitely check the trace. Also, one strange thing I noticed, with number of threads till '64', the latency is around 3ms but as the number of threads increases latency also increases. Eventually, at thread count, 609 latency is around 30ms. I am using a single client to loadtes

Re: Increased latency after setting row_cache_size_in_mb

2018-02-05 Thread Jeff Jirsa
Also: coordinator handles tracing and read repair. Make sure tracing is off for production. Have your data repaired if possible to eliminate that. Use tracing to see what’s taking the time. -- Jeff Jirsa > On Feb 5, 2018, at 6:32 AM, Jeff Jirsa wrote: > > There’s two parts to latency on the

Re: Increased latency after setting row_cache_size_in_mb

2018-02-05 Thread Jeff Jirsa
There’s two parts to latency on the Cassandra side: Local and coordinator When you read, the node to which you connect coordinates the request to the node which has the data (potentially itself). Long tail in coordinator latencies tend to be the coordinator itself gc’ing, which will happen from

Re: Increased latency after setting row_cache_size_in_mb

2018-02-05 Thread Rahul Singh
What are the tbl Local read latency stats vs. the read request latency stats ? Rahul On Feb 5, 2018, 8:41 AM -0500, mohsin k , wrote: > Thanks for response @Nicolas. I was considering the total read latency from > the client to server (as shown in the image above) which is around 30ms. > Which

Re: Increased latency after setting row_cache_size_in_mb

2018-02-05 Thread mohsin k
Thanks for response @Nicolas. I was considering the total read latency from the client to server (as shown in the image above) which is around 30ms. Which I want to get around 3ms (client and server are both on same network). I did not consider read latency provided by the server (which I should ha

Re: Increased latency after setting row_cache_size_in_mb

2018-02-05 Thread Nicolas Guyomar
Your row hit rate is 0.971 which is already very high, IMHO there is "nothing" left to do here if you can afford to store your entire dataset in memory Local read latency: 0.030 ms already seems good to me, what makes you think that you can achieve more with the relative "small" box you are using

Re: Increased latency after setting row_cache_size_in_mb

2018-02-05 Thread mohsin k
Hey Rahul, Each partition has around 10 cluster keys. Based on nodetool, I can roughly estimate partition size to be less than 1KB. On Mon, Feb 5, 2018 at 5:37 PM, mohsin k wrote: > Hey Nicolas, > > My goal is to reduce latency as much as possible. I did wait for warmup. > The test ran for more

Re: Increased latency after setting row_cache_size_in_mb

2018-02-05 Thread mohsin k
Hey Nicolas, My goal is to reduce latency as much as possible. I did wait for warmup. The test ran for more than 15mins, I am not sure why it shows 2mins though. On Mon, Feb 5, 2018 at 5:25 PM, Rahul Singh wrote: > What is the average size of your partitions / rows. 1GB may not be enough. > >

Re: Increased latency after setting row_cache_size_in_mb

2018-02-05 Thread Rahul Singh
What is the average size of your partitions / rows. 1GB may not be enough. Rahul On Feb 5, 2018, 6:52 AM -0500, mohsin k , wrote: > Hi, > > I have been looking into different configurations for tuning my cassandra > servers. So, initially I loadtested server using cassandra-stress tool, with >

Re: Increased latency after setting row_cache_size_in_mb

2018-02-05 Thread Nicolas Guyomar
Hi, Could you explain a bit more what you are trying to achieve please ? Performance tuning is by far the most complex problem we have to deal with, and there are a lot of configuration changes that can be made on a C* cluster When you do performance tuning, do not forget that you need to warmup