One other thing to look at besides the heap is your commit settings. We've experienced something similar, and changing commit settings alleviated the issue.
Are you opening a search on every hardcommit? If so, you might want to reconsider and use the softcommit for the hourly creation of a new searcher. The hardcommit interval should be much lower. Probably something on the order of seconds (15000) instead of hours (currently 3600000). When the hard commit fires, numerous merges might be firing off in the background due to the volume of documents you are indexing, which might explain the periodic bad response times shown in your logs. It would depend on your specific scenario, but here's our setup. During long periods of constant indexing of documents to a staging collection (~2 billion documents), we have following commit settings softcommit: 3600000ms (for periodic validation of data, since it's not in production) hardcommit: openSearcher -> false, 15000ms (no document limit) This makes the documents available for searching every hour, but doesn't result in the large bursts of IO due to the infrequent hard commits. For more info, Erick Erickson has a great write up: https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Best, Chris On Mon, Mar 18, 2019 at 9:36 AM Aaron Yingcai Sun <y...@vizrt.com> wrote: > Hi, Emir, > > My system used to run with max 32GB, the response time is bad as well. > swap is set to 4GB, there 3.2 free, I doubt swap would affect it since > there is such huge free memory. > > I could try to with set Xms and Xmx to the same value, but I doubt how > much would that change the response time. > > > BRs > > //Aaron > > ________________________________ > From: Emir Arnautović <emir.arnauto...@sematext.com> > Sent: Monday, March 18, 2019 2:19:19 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr index slow response > > Hi Aaron, > Without looking too much into numbers, my bet would be that it is large > heap that is causing issues. I would decrease is significantly (<30GB) and > see if it is enough for your max load. Also, disable swap or reduce > swappiness to min. > > In any case, you should install some monitoring tool that would help you > do better analysis when you run into problems. One such tool is our > monitoring solution: https://sematext.com/spm > > HTH, > Emir > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > > > On 18 Mar 2019, at 13:14, Aaron Yingcai Sun <y...@vizrt.com> wrote: > > > > Hello, Emir, > > > > Thanks for the reply, this is the solr version and heap info, standalone > single solr server. I don't have monitor tool connected. only look at > 'top', has not seen cpu spike so far, when the slow response happens, cpu > usage is not high at all, around 30%. > > > > > > # curl 'http://.../solr/admin/info/system?wt=json&indent=true' > > { > > "responseHeader":{ > > "status":0, > > "QTime":27}, > > "mode":"std", > > "solr_home":"/ardome/solr", > > "lucene":{ > > "solr-spec-version":"6.5.1", > > "solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - > jimczi - 2017-04-21 12:23:42", > > "lucene-spec-version":"6.5.1", > > "lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 > - jimczi - 2017-04-21 12:17:15"}, > > "jvm":{ > > "version":"1.8.0_144 25.144-b01", > > "name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM", > > "spec":{ > > "vendor":"Oracle Corporation", > > "name":"Java Platform API Specification", > > "version":"1.8"}, > > "jre":{ > > "vendor":"Oracle Corporation", > > "version":"1.8.0_144"}, > > "vm":{ > > "vendor":"Oracle Corporation", > > "name":"Java HotSpot(TM) 64-Bit Server VM", > > "version":"25.144-b01"}, > > "processors":32, > > "memory":{ > > "free":"69.1 GB", > > "total":"180.2 GB", > > "max":"266.7 GB", > > "used":"111 GB (%41.6)", > > "raw":{ > > "free":74238728336, > > "total":193470136320, > > "max":286331502592, > > "used":119231407984, > > "used%":41.64103736566334}}, > > "jmx":{ > > > "bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes", > > "classpath":"...", > > "commandLineArgs":["-Xms100G", > > "-Xmx300G", > > "-DSTOP.PORT=8079", > > "-DSTOP.KEY=..", > > "-Dsolr.solr.home=..", > > "-Djetty.port=8983"], > > "startTime":"2019-03-18T09:35:27.892Z", > > "upTimeMS":9258422}}, > > "system":{ > > "name":"Linux", > > "arch":"amd64", > > "availableProcessors":32, > > "systemLoadAverage":14.72, > > "version":"3.0.101-311.g08a8a9d-default", > > "committedVirtualMemorySize":2547960700928, > > "freePhysicalMemorySize":4530696192, > > "freeSwapSpaceSize":3486846976, > > "processCpuLoad":0.3257436126790475, > > "processCpuTime":93869450000000, > > "systemCpuLoad":0.3279781055816521, > > "totalPhysicalMemorySize":406480175104, > > "totalSwapSpaceSize":4302303232 <(430)%20230-3232>, > > "maxFileDescriptorCount":32768, > > "openFileDescriptorCount":385, > > "uname":"Linux ... 3.0.101-311.g08a8a9d-default #1 SMP Wed Dec 14 > 10:15:37 UTC 2016 (08a8a9d) x86_64 x86_64 x86_64 GNU/Linux\n", > > "uptime":" 13:09pm up 5 days 21:23, 7 users, load average: 14.72, > 12.28, 11.48\n"}} > > > > > > > > > > ________________________________ > > From: Emir Arnautović <emir.arnauto...@sematext.com> > > Sent: Monday, March 18, 2019 12:10:30 PM > > To: solr-user@lucene.apache.org > > Subject: Re: Solr index slow response > > > > Hi Aaron, > > Which version of Solr? How did you configure your heap? Is it standalone > Solr or SolrCloud? A single server? Do you use some monitoring tool? Do you > see some spikes, pauses or CPU usage is constant? > > > > Thanks, > > Emir > > -- > > Monitoring - Log Management - Alerting - Anomaly Detection > > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > > > > > >> On 18 Mar 2019, at 11:47, Aaron Yingcai Sun <y...@vizrt.com> wrote: > >> > >> Hello, Solr! > >> > >> > >> We are having some performance issue when try to send documents for > solr to index. The repose time is very slow and unpredictable some time. > >> > >> > >> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, > while 300 GB is reserved for solr, while this happening, cpu usage is > around 30%, mem usage is 34%. io also look ok according to iotop. SSD disk. > >> > >> > >> Our application send 100 documents to solr per request, json encoded. > the size is around 5M each time. some times the response time is under 1 > seconds, some times could be 300 seconds, the slow response happens very > often. > >> > >> > >> "Soft AutoCommit: disabled", "Hard AutoCommit: if uncommited for > 3600000ms; if 1000000 uncommited docs" > >> > >> > >> There are around 100 clients sending those documents at the same time, > but each for the client is blocking call which wait the http response then > send the next one. > >> > >> > >> I tried to make the number of documents smaller in one request, such as > 20, but still I see slow response time to time, like 80 seconds. > >> > >> > >> Would you help to give some hint how improve the response time? solr > does not seems very loaded, there must be a way to make the response faster. > >> > >> > >> BRs > >> > >> //Aaron > >> > >> > >> > > > >
