I'd check my I/O. Since you're firing the same query, I expect that you aren't I/O bound at all, since, as you say, the docs should already be in memory. This assumes that your document cache size is > 0. You can check this. Go to the admin UI, select one of your cores (not collection) and go to plugins/stats. You should see the documentCache as one of the entries and you should be hitting an insane hit ratio close to 100% as your test runs.
Also check your queryResultCache. That also should be near 100% in your test. Do note that these caches really never hit this "for real", but as you indicated this is a highly artificial test so such high hit ratios are what I'd expect. Assuming that those caches are being hit near 100%, Solr really isn't doing any work to speak of so there almost has to be some kind of queueing going on. The fact that your CPU is only running 8-10% is an indication that you're requests are queued up somewhere, but where I have no clue. The Jetty thread pool is quite large. What are the QTimes reported in the responses? My guess is that the QTime stays pretty constant (and very low) even as your response time increases, another indication that you're queueing. Hmmm, is it possible that on the queueing is on the _client_ side? What aggregate throughput to you get if you fire up 10 _clients_ each with one thread rather than 1 client and 10 threads? That's a shot in the dark, but worth a try I suppose. And how does your client fire queries? SolrJ? Http? Jmeter or the like? But yeah, this is weird. Since you're firing the same query, Solr isn't really doing any work at all. Best, Erick On Thu, Mar 23, 2017 at 7:56 PM, Aman Deep Singh <amandeep.coo...@gmail.com> wrote: > You can play with the merge factor in the index config. > If their is no frequent updates then make it 2 ,it will give you High > throughput and less latency. > > On 24-Mar-2017 8:22 AM, "Zheng Lin Edwin Yeo" <edwinye...@gmail.com> wrote: > >> I also did find that beyond 10 threads for 8GB heap size , there isn't much >> improvement with the performance. But you can increase your heap size a >> little if your system allows it. >> >> By the way, which Solr version are you using? >> >> Regards, >> Edwin >> >> >> On 24 March 2017 at 09:21, Matt Magnusson <magnuss...@gmail.com> wrote: >> >> > Out of curosity, what is your index size? I'm trying to do something >> > similar with maximizing output, I'm currently looking at streaming >> > expressions which I'm seeing some interesting results for, I'm also >> > finding that the direct mass query route seems to hit a wall for >> > performance. I'm also finding that about 10 threads seems to be an >> > optimum number. >> > >> > On Thu, Mar 23, 2017 at 8:10 PM, Suresh Pendap <spen...@walmartlabs.com> >> > wrote: >> > > Hi, >> > > I am new to SOLR search engine technology and I am trying to get some >> > performance numbers to get maximum throughput from the SOLR cluster of a >> > given size. >> > > I am currently doing only query load testing in which I randomly fire a >> > bunch of queries to the SOLR cluster to generate the query load. I >> > understand that it is not the ideal workload as the >> > > ingestion and commits happening invalidate the Solr Caches, so it is >> > advisable to perform query load along with some documents being ingested. >> > > >> > > The SOLR cluster was made up of 2 shards and 2 replicas. So there were >> > total 4 replicas serving the queries. The SOLR nodes were running on an >> LXD >> > container with 12 cores and 88GB RAM. >> > > The heap size allocated was 8g min and 8g max. All the other SOLR >> > configurations were default. >> > > >> > > The client node was running on an 8 core VM. >> > > >> > > I performed the test with 1 thread, 10 client threads and 50 client >> > threads. I noticed that as I increased the number of threads, the query >> > latency kept increasing drastically which I was not expecting. >> > > >> > > Since my initial test was randomly picking queries from a file, I >> > decided to keep things constant and ran the program which fired the same >> > query again and again. Since it is the same query, all the documents will >> > > be in the Cache and the query response time should be very fast. I was >> > also expecting that with 10 or 50 client threads, the query latency >> should >> > not be increasing. >> > > >> > > The throughput increased only up to 10 client threads but then it was >> > same for 50 threads, 100 threads and the latency of the query kept >> > increasing as I increased the number of threads. >> > > The query was returning 2 documents only. >> > > >> > > The table below summarizes the numbers that I was saying with a single >> > query. >> > > >> > > >> > > >> > > >> > > >> > > #No of Client Nodes >> > > #No of Threads 99 pct Latency 95 pct latency throughput >> > CPU Utilization Server Configuration >> > > >> > > 1 1 9 ms 7 ms 180 reqs/sec 8% >> > > >> > > Heap size: ms=8g, mx=8g >> > > >> > > default configuration >> > > >> > > >> > > 1 10 400 ms 360 ms 360 reqs/sec 10% >> > > >> > > Heap size: ms=8g, mx=8g >> > > >> > > default configuration >> > > >> > > >> > > >> > > >> > > I also ran the client program on the SOLR server node in order to rule >> > our the network latency factor. On the server node also the response time >> > was higher for 10 threads, but the amplification was smaller. >> > > >> > > I am getting an impression that probably my query requests are getting >> > queued up and limited due to probably some thread pool size on the server >> > side. However I saw that the default jetty.xml does >> > > have the thread pool of min size of 10 and max of 10000. >> > > >> > > Is there any other internal SOLR thread pool configuration which might >> > be limiting the query response time? >> > > >> > > I wanted to check with the community if what I am seeing is abnormal >> > behavior, what could be the issue? Is there any configuration that I can >> > tweak to get better query response times for more load? >> > > >> > > Regards >> > > Suresh >> > > >> > >>
