Re: Throughput does not increase in spite of low CPU usage

bq. My customer requested me to achieve 1000 qps with a single Solr. You need to talk to your customer then and adjust expectations. You say you get 60 QPS with 40% CPU utilization. So let’s say QPS is perfectly linear to CPU utilization and you can get CPU utilization to run at 100%. In that c

Re: Throughput does not increase in spite of low CPU usage

On Tue, 2019-10-01 at 19:08 +0900, Yasufumi Mizoguchi wrote: > * The number of Firing hosts : 6 > * [Each host]ThreadGroup.num_threads : 200 > * [Each host]ThreadGroup.ramp_time : 600 > * [Each host]ThreadGroup.duration: 1800 > * [Each host]ThreadGroup.delay: 0 > * The number of sample queries: 20,

Re: Throughput does not increase in spite of low CPU usage

Hi Paras, Thank you for your advice. I will confirm JMeter's settings in addition to JVM options. Yes, we are using documentCache and *after* finishing load test, we will comment that out. (By customer requests, we cannot update cache settings till the test ends...) Thanks, Yasufumi 2019年10月1日

Re: Throughput does not increase in spite of low CPU usage

It is difficult to answer that for me. My customer requested me to achieve 1000 qps with single Solr. Thanks, Yasufumi. 2019年10月1日(火) 14:59 Jörn Franke : > Why do you need 1000 qps? > > > > Am 30.09.2019 um 07:45 schrieb Yasufumi Mizoguchi < > yasufumi0...@gmail.com>: > > > > Hi, > > > > I am

Re: Throughput does not increase in spite of low CPU usage

Hi Yasufumi, Followings are current load test set up. Did you try decreasing ramp_time and increasing num_threads or Firing hosts? Out of 1800 secs, you are giving 200 threads a maximum of 600 seconds to get ready. I don't increase with this value when I want to test parallel requests. In your c

Re: Throughput does not increase in spite of low CPU usage

Thank you for replying me. I will try to resize NewRatio. Thanks, Yasufumi. 2019年10月1日(火) 11:19 Deepak Goel : > Hello > > Can you please try increasing 'new size' and 'max new size' to 1GB+? > > Deepak > > On Mon, 30 Sep 2019, 13:35 Yasufumi Mizoguchi, > wrote: > > > Hi, Deepak. > > Thank you

Re: Throughput does not increase in spite of low CPU usage

Thank you for your response. Now, we have no JVM monitoring. But checking GC logs, I found no major GC during load test. As you saying, heap size might be too large and I am planning to reduce that. Thanks, Yasufumi 2019年9月30日(月) 23:19 Walter Underwood : > 31G is still a very large heap. We use

Re: Throughput does not increase in spite of low CPU usage

Ah, sorry. Not JUnit, we use JMeter. Thanks, Yasufumi 2019年10月1日(火) 19:08 Yasufumi Mizoguchi : > Thank you for replying me. > > Followings are current load test set up. > > * Load test program : JUnit > * The number of Firing hosts : 6 > * [Each host]ThreadGroup.num_threads : 200 > * [Each host

Re: Throughput does not increase in spite of low CPU usage

Thank you for your response. Our index has about 25,000,000 docs. Is this quite difficult to achieve 1000 qps with single Solr instance even if all index files are cached on OS page cache? And as you pointed out, there are lots of minor GC during the test, I will try to resize NewRatio. Thanks,

Re: Throughput does not increase in spite of low CPU usage

Thank you for replying me. Followings are current load test set up. * Load test program : JUnit * The number of Firing hosts : 6 * [Each host]ThreadGroup.num_threads : 200 * [Each host]ThreadGroup.ramp_time : 600 * [Each host]ThreadGroup.duration: 1800 * [Each host]ThreadGroup.delay: 0 * The numb

Re: Throughput does not increase in spite of low CPU usage

Besides what all have suggested, can you share your testing setup? Are you using JMeter? I'm asking this to confirm that your setup is "actually trying" to generate 1000 simultaneous threads and not bottlenecking the process. I get 170 qps easily on a simpler server but I remember that I could not

Re: Throughput does not increase in spite of low CPU usage

Why do you need 1000 qps? > Am 30.09.2019 um 07:45 schrieb Yasufumi Mizoguchi : > > Hi, > > I am trying some tests to confirm if single Solr instance can perform over > 1000 queries per second(!). > > But now, although CPU usage is 40% or so and iowait is almost 0%, > throughput does not incr

Re: Throughput does not increase in spite of low CPU usage

Hello Can you please try increasing 'new size' and 'max new size' to 1GB+? Deepak On Mon, 30 Sep 2019, 13:35 Yasufumi Mizoguchi, wrote: > Hi, Deepak. > Thank you for replying me. > > JVM settings from solr.in.sh file are as follows. (Sorry, I could not show > all due to our policy) > > -verbos

Re: Throughput does not increase in spite of low CPU usage

31G is still a very large heap. We use 8G for all of our different clusters. Do you have JVM monitoring? Look at the heap used after a major GC. Use that number, plus some extra, for the heap size. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Sep

Re: Throughput does not increase in spite of low CPU usage

On 9/29/2019 11:44 PM, Yasufumi Mizoguchi wrote: I am trying some tests to confirm if single Solr instance can perform over 1000 queries per second(!). In general, I would never expect a single instance to handle a large number of queries per second unless the index is REALLY small -- dozens

Re: Throughput does not increase in spite of low CPU usage

The most basic question is how you are load-testing it? Assuming you have some kind of client firing queries at Solr, keep adding threads so Solr is handling more and more queries in parallel. If you start to see the response time at the client get longer _and_ the QTime in Solr’s response stay

Re: Throughput does not increase in spite of low CPU usage

Hi, Ere. Thank you for valuable feedback. I will try Xmx31G and Xms31G instead of current ones. Thanks and Regards, Yasufumi. 2019年9月30日(月) 17:19 Ere Maijala : > Just a side note: -Xmx32G is really bad for performance as it forces > Java to use non-compressed pointers. You'll actually get bette

Re: Throughput does not increase in spite of low CPU usage

Just a side note: -Xmx32G is really bad for performance as it forces Java to use non-compressed pointers. You'll actually get better results with -Xmx31G. For more information, see e.g. https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/ Regards, Ere Yasufumi Mizo

Re: Throughput does not increase in spite of low CPU usage

Hi, Deepak. Thank you for replying me. JVM settings from solr.in.sh file are as follows. (Sorry, I could not show all due to our policy) -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTi

Re: Throughput does not increase in spite of low CPU usage

Hello Can you please share the JVM heap settings in detail? Deepak On Mon, 30 Sep 2019, 11:15 Yasufumi Mizoguchi, wrote: > Hi, > > I am trying some tests to confirm if single Solr instance can perform over > 1000 queries per second(!). > > But now, although CPU usage is 40% or so and iowait is

Throughput does not increase in spite of low CPU usage

Hi, I am trying some tests to confirm if single Solr instance can perform over 1000 queries per second(!). But now, although CPU usage is 40% or so and iowait is almost 0%, throughput does not increase over 60 queries per second. I think there are some bottlenecks around Kernel, JVM, or Solr set