One more thing - it is considered a good practice to use the same value for Xmx and Xms.
Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 18 Mar 2019, at 14:19, Emir Arnautović <emir.arnauto...@sematext.com> > wrote: > > Hi Aaron, > Without looking too much into numbers, my bet would be that it is large heap > that is causing issues. I would decrease is significantly (<30GB) and see if > it is enough for your max load. Also, disable swap or reduce swappiness to > min. > > In any case, you should install some monitoring tool that would help you do > better analysis when you run into problems. One such tool is our monitoring > solution: https://sematext.com/spm > > HTH, > Emir > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > >> On 18 Mar 2019, at 13:14, Aaron Yingcai Sun <y...@vizrt.com> wrote: >> >> Hello, Emir, >> >> Thanks for the reply, this is the solr version and heap info, standalone >> single solr server. I don't have monitor tool connected. only look at 'top', >> has not seen cpu spike so far, when the slow response happens, cpu usage is >> not high at all, around 30%. >> >> >> # curl 'http://.../solr/admin/info/system?wt=json&indent=true' >> { >> "responseHeader":{ >> "status":0, >> "QTime":27}, >> "mode":"std", >> "solr_home":"/ardome/solr", >> "lucene":{ >> "solr-spec-version":"6.5.1", >> "solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - >> jimczi - 2017-04-21 12:23:42", >> "lucene-spec-version":"6.5.1", >> "lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - >> jimczi - 2017-04-21 12:17:15"}, >> "jvm":{ >> "version":"1.8.0_144 25.144-b01", >> "name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM", >> "spec":{ >> "vendor":"Oracle Corporation", >> "name":"Java Platform API Specification", >> "version":"1.8"}, >> "jre":{ >> "vendor":"Oracle Corporation", >> "version":"1.8.0_144"}, >> "vm":{ >> "vendor":"Oracle Corporation", >> "name":"Java HotSpot(TM) 64-Bit Server VM", >> "version":"25.144-b01"}, >> "processors":32, >> "memory":{ >> "free":"69.1 GB", >> "total":"180.2 GB", >> "max":"266.7 GB", >> "used":"111 GB (%41.6)", >> "raw":{ >> "free":74238728336, >> "total":193470136320, >> "max":286331502592, >> "used":119231407984, >> "used%":41.64103736566334}}, >> "jmx":{ >> >> "bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes", >> "classpath":"...", >> "commandLineArgs":["-Xms100G", >> "-Xmx300G", >> "-DSTOP.PORT=8079", >> "-DSTOP.KEY=..", >> "-Dsolr.solr.home=..", >> "-Djetty.port=8983"], >> "startTime":"2019-03-18T09:35:27.892Z", >> "upTimeMS":9258422}}, >> "system":{ >> "name":"Linux", >> "arch":"amd64", >> "availableProcessors":32, >> "systemLoadAverage":14.72, >> "version":"3.0.101-311.g08a8a9d-default", >> "committedVirtualMemorySize":2547960700928, >> "freePhysicalMemorySize":4530696192, >> "freeSwapSpaceSize":3486846976, >> "processCpuLoad":0.3257436126790475, >> "processCpuTime":93869450000000, >> "systemCpuLoad":0.3279781055816521, >> "totalPhysicalMemorySize":406480175104, >> "totalSwapSpaceSize":4302303232, >> "maxFileDescriptorCount":32768, >> "openFileDescriptorCount":385, >> "uname":"Linux ... 3.0.101-311.g08a8a9d-default #1 SMP Wed Dec 14 10:15:37 >> UTC 2016 (08a8a9d) x86_64 x86_64 x86_64 GNU/Linux\n", >> "uptime":" 13:09pm up 5 days 21:23, 7 users, load average: 14.72, >> 12.28, 11.48\n"}} >> >> >> >> >> ________________________________ >> From: Emir Arnautović <emir.arnauto...@sematext.com> >> Sent: Monday, March 18, 2019 12:10:30 PM >> To: solr-user@lucene.apache.org >> Subject: Re: Solr index slow response >> >> Hi Aaron, >> Which version of Solr? How did you configure your heap? Is it standalone >> Solr or SolrCloud? A single server? Do you use some monitoring tool? Do you >> see some spikes, pauses or CPU usage is constant? >> >> Thanks, >> Emir >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >> >> >> >>> On 18 Mar 2019, at 11:47, Aaron Yingcai Sun <y...@vizrt.com> wrote: >>> >>> Hello, Solr! >>> >>> >>> We are having some performance issue when try to send documents for solr to >>> index. The repose time is very slow and unpredictable some time. >>> >>> >>> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, while >>> 300 GB is reserved for solr, while this happening, cpu usage is around 30%, >>> mem usage is 34%. io also look ok according to iotop. SSD disk. >>> >>> >>> Our application send 100 documents to solr per request, json encoded. the >>> size is around 5M each time. some times the response time is under 1 >>> seconds, some times could be 300 seconds, the slow response happens very >>> often. >>> >>> >>> "Soft AutoCommit: disabled", "Hard AutoCommit: if uncommited for 3600000ms; >>> if 1000000 uncommited docs" >>> >>> >>> There are around 100 clients sending those documents at the same time, but >>> each for the client is blocking call which wait the http response then send >>> the next one. >>> >>> >>> I tried to make the number of documents smaller in one request, such as 20, >>> but still I see slow response time to time, like 80 seconds. >>> >>> >>> Would you help to give some hint how improve the response time? solr does >>> not seems very loaded, there must be a way to make the response faster. >>> >>> >>> BRs >>> >>> //Aaron >>> >>> >>> >> >
