I'm a bit confused, why large heap size would make it slower? Isn't that give it enough room to make it not busy doing GC all the time?
My http/json request contains 100 documents, the total size of the 100 documents is around 5M, there are ~100 client sending those requests continuously. Previously the JVM is set to max 32 GB , the speed was even worse, now it's running with min 100GB, max 300GB, it use around 100GB. this page suggest to use smaller number of documents per request, https://wiki.apache.org/solr/SolrPerformanceProblems SolrPerformanceProblems - Solr Wiki<https://wiki.apache.org/solr/SolrPerformanceProblems> wiki.apache.org General information. There is a performance bug that makes *everything* slow in versions 6.4.0 and 6.4.1. The problem is fixed in 6.4.2. It is described by SOLR-10130.This is highly version specific, so if you are not running one of the affected versions, don't worry about it. So I try to reduce the number, still I could get lots of large response QTime: 190318-142652.695-160214 DBG1:doc_count: 10 , doc_size: 609 KB, Res code: 200, QTime: 47918 ms, Request time: 47921 ms. 190318-142652.704-160179 DBG1:doc_count: 10 , doc_size: 568 KB, Res code: 200, QTime: 36919 ms, Request time: 36922 ms. 190318-142652.780-160197 DBG1:doc_count: 10 , doc_size: 609 KB, Res code: 200, QTime: 36082 ms, Request time: 36084 ms. 190318-142652.859-160200 DBG1:doc_count: 10 , doc_size: 569 KB, Res code: 200, QTime: 36880 ms, Request time: 36882 ms. 190318-142653.131-160148 DBG1:doc_count: 10 , doc_size: 608 KB, Res code: 200, QTime: 37222 ms, Request time: 37224 ms. 190318-142653.154-160211 DBG1:doc_count: 10 , doc_size: 541 KB, Res code: 200, QTime: 37241 ms, Request time: 37243 ms. 190318-142653.223-163490 DBG1:doc_count: 10 , doc_size: 589 KB, Res code: 200, QTime: 37174 ms, Request time: 37176 ms. 190318-142653.359-160154 DBG1:doc_count: 10 , doc_size: 592 KB, Res code: 200, QTime: 37008 ms, Request time: 37011 ms. 190318-142653.497-163491 DBG1:doc_count: 10 , doc_size: 583 KB, Res code: 200, QTime: 24828 ms, Request time: 24830 ms. 190318-142653.987-160208 DBG1:doc_count: 10 , doc_size: 669 KB, Res code: 200, QTime: 23900 ms, Request time: 23902 ms. 190318-142654.114-160208 DBG1:doc_count: 10 , doc_size: 544 KB, Res code: 200, QTime: 121 ms, Request time: 122 ms. 190318-142654.233-160208 DBG1:doc_count: 10 , doc_size: 536 KB, Res code: 200, QTime: 113 ms, Request time: 115 ms. 190318-142654.354-160208 DBG1:doc_count: 10 , doc_size: 598 KB, Res code: 200, QTime: 116 ms, Request time: 117 ms. 190318-142654.466-160208 DBG1:doc_count: 10 , doc_size: 546 KB, Res code: 200, QTime: 107 ms, Request time: 108 ms. 190318-142654.586-160208 DBG1:doc_count: 10 , doc_size: 566 KB, Res code: 200, QTime: 114 ms, Request time: 115 ms. 190318-142654.687-160208 DBG1:doc_count: 10 , doc_size: 541 KB, Res code: 200, QTime: 96 ms, Request time: 98 ms. 190318-142654.768-160208 DBG1:doc_count: 10 , doc_size: 455 KB, Res code: 200, QTime: 75 ms, Request time: 77 ms. 190318-142654.870-160208 DBG1:doc_count: 10 , doc_size: 538 KB, Res code: 200, QTime: 97 ms, Request time: 98 ms. 190318-142654.967-160208 DBG1:doc_count: 10 , doc_size: 539 KB, Res code: 200, QTime: 92 ms, Request time: 93 ms. 190318-142655.096-160208 DBG1:doc_count: 10 , doc_size: 672 KB, Res code: 200, QTime: 124 ms, Request time: 125 ms. 190318-142655.210-160208 DBG1:doc_count: 10 , doc_size: 605 KB, Res code: 200, QTime: 108 ms, Request time: 110 ms. 190318-142655.304-160208 DBG1:doc_count: 10 , doc_size: 481 KB, Res code: 200, QTime: 89 ms, Request time: 90 ms. 190318-142655.410-160208 DBG1:doc_count: 10 , doc_size: 468 KB, Res code: 200, QTime: 101 ms, Request time: 102 ms. ________________________________ From: Toke Eskildsen <t...@kb.dk> Sent: Monday, March 18, 2019 2:13:12 PM To: solr-user@lucene.apache.org Subject: Re: Solr index slow response On Mon, 2019-03-18 at 10:47 +0000, Aaron Yingcai Sun wrote: > Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, > while 300 GB is reserved for solr, [...] 300GB for Solr sounds excessive. > Our application send 100 documents to solr per request, json encoded. > the size is around 5M each time. some times the response time is > under 1 seconds, some times could be 300 seconds, the slow response > happens very often. > ... > There are around 100 clients sending those documents at the same > time, but each for the client is blocking call which wait the http > response then send the next one. 100 clients * 5MB/batch = at most 500MB. Or maybe you meant 100 clients * 100 documents * 5MB/document = at most 50GB? Either way it is a long way from 300GB and the stats you provide further down the thread indicates that you are overprovisioning quite a lot: "memory":{ "free":"69.1 GB", "total":"180.2 GB", "max":"266.7 GB", "used":"111 GB (%41.6)", Intermittent slow response times are a known effect of having large Java heaps, due to stop-the-world garbage collections. Try dialing Xmx _way_ down: If your batches are only 5MB each, try Xmx=20g or less. I know that the stats above says that Solr uses 111GB, but the JVM has a tendency to expand the heap quite a lot when it is getting hammered. If you want to check beforehand, you can see how much memeory is freed from full GCs in the GC-log. - Toke Eskildsen, Royal Danish Library
