4GB swap on 400GB machine does not make much sense, so disable it. Even 4GB, some pages might be swapped, and if those are some Solr pages, it’ll affect Solr.
Setting Xms and Xmx to the same value will not solve your issue but you will avoid heap resize when your heap reaches Xms. Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 18 Mar 2019, at 14:36, Aaron Yingcai Sun <y...@vizrt.com> wrote: > > Hi, Emir, > > My system used to run with max 32GB, the response time is bad as well. swap > is set to 4GB, there 3.2 free, I doubt swap would affect it since there is > such huge free memory. > > I could try to with set Xms and Xmx to the same value, but I doubt how much > would that change the response time. > > > BRs > > //Aaron > > ________________________________ > From: Emir Arnautović <emir.arnauto...@sematext.com> > Sent: Monday, March 18, 2019 2:19:19 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr index slow response > > Hi Aaron, > Without looking too much into numbers, my bet would be that it is large heap > that is causing issues. I would decrease is significantly (<30GB) and see if > it is enough for your max load. Also, disable swap or reduce swappiness to > min. > > In any case, you should install some monitoring tool that would help you do > better analysis when you run into problems. One such tool is our monitoring > solution: https://sematext.com/spm > > HTH, > Emir > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > >> On 18 Mar 2019, at 13:14, Aaron Yingcai Sun <y...@vizrt.com> wrote: >> >> Hello, Emir, >> >> Thanks for the reply, this is the solr version and heap info, standalone >> single solr server. I don't have monitor tool connected. only look at 'top', >> has not seen cpu spike so far, when the slow response happens, cpu usage is >> not high at all, around 30%. >> >> >> # curl 'http://.../solr/admin/info/system?wt=json&indent=true' >> { >> "responseHeader":{ >> "status":0, >> "QTime":27}, >> "mode":"std", >> "solr_home":"/ardome/solr", >> "lucene":{ >> "solr-spec-version":"6.5.1", >> "solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - >> jimczi - 2017-04-21 12:23:42", >> "lucene-spec-version":"6.5.1", >> "lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - >> jimczi - 2017-04-21 12:17:15"}, >> "jvm":{ >> "version":"1.8.0_144 25.144-b01", >> "name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM", >> "spec":{ >> "vendor":"Oracle Corporation", >> "name":"Java Platform API Specification", >> "version":"1.8"}, >> "jre":{ >> "vendor":"Oracle Corporation", >> "version":"1.8.0_144"}, >> "vm":{ >> "vendor":"Oracle Corporation", >> "name":"Java HotSpot(TM) 64-Bit Server VM", >> "version":"25.144-b01"}, >> "processors":32, >> "memory":{ >> "free":"69.1 GB", >> "total":"180.2 GB", >> "max":"266.7 GB", >> "used":"111 GB (%41.6)", >> "raw":{ >> "free":74238728336, >> "total":193470136320, >> "max":286331502592, >> "used":119231407984, >> "used%":41.64103736566334}}, >> "jmx":{ >> >> "bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes", >> "classpath":"...", >> "commandLineArgs":["-Xms100G", >> "-Xmx300G", >> "-DSTOP.PORT=8079", >> "-DSTOP.KEY=..", >> "-Dsolr.solr.home=..", >> "-Djetty.port=8983"], >> "startTime":"2019-03-18T09:35:27.892Z", >> "upTimeMS":9258422}}, >> "system":{ >> "name":"Linux", >> "arch":"amd64", >> "availableProcessors":32, >> "systemLoadAverage":14.72, >> "version":"3.0.101-311.g08a8a9d-default", >> "committedVirtualMemorySize":2547960700928, >> "freePhysicalMemorySize":4530696192, >> "freeSwapSpaceSize":3486846976, >> "processCpuLoad":0.3257436126790475, >> "processCpuTime":93869450000000, >> "systemCpuLoad":0.3279781055816521, >> "totalPhysicalMemorySize":406480175104, >> "totalSwapSpaceSize":4302303232, >> "maxFileDescriptorCount":32768, >> "openFileDescriptorCount":385, >> "uname":"Linux ... 3.0.101-311.g08a8a9d-default #1 SMP Wed Dec 14 10:15:37 >> UTC 2016 (08a8a9d) x86_64 x86_64 x86_64 GNU/Linux\n", >> "uptime":" 13:09pm up 5 days 21:23, 7 users, load average: 14.72, >> 12.28, 11.48\n"}} >> >> >> >> >> ________________________________ >> From: Emir Arnautović <emir.arnauto...@sematext.com> >> Sent: Monday, March 18, 2019 12:10:30 PM >> To: solr-user@lucene.apache.org >> Subject: Re: Solr index slow response >> >> Hi Aaron, >> Which version of Solr? How did you configure your heap? Is it standalone >> Solr or SolrCloud? A single server? Do you use some monitoring tool? Do you >> see some spikes, pauses or CPU usage is constant? >> >> Thanks, >> Emir >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >> >> >> >>> On 18 Mar 2019, at 11:47, Aaron Yingcai Sun <y...@vizrt.com> wrote: >>> >>> Hello, Solr! >>> >>> >>> We are having some performance issue when try to send documents for solr to >>> index. The repose time is very slow and unpredictable some time. >>> >>> >>> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, while >>> 300 GB is reserved for solr, while this happening, cpu usage is around 30%, >>> mem usage is 34%. io also look ok according to iotop. SSD disk. >>> >>> >>> Our application send 100 documents to solr per request, json encoded. the >>> size is around 5M each time. some times the response time is under 1 >>> seconds, some times could be 300 seconds, the slow response happens very >>> often. >>> >>> >>> "Soft AutoCommit: disabled", "Hard AutoCommit: if uncommited for 3600000ms; >>> if 1000000 uncommited docs" >>> >>> >>> There are around 100 clients sending those documents at the same time, but >>> each for the client is blocking call which wait the http response then send >>> the next one. >>> >>> >>> I tried to make the number of documents smaller in one request, such as 20, >>> but still I see slow response time to time, like 80 seconds. >>> >>> >>> Would you help to give some hint how improve the response time? solr does >>> not seems very loaded, there must be a way to make the response faster. >>> >>> >>> BRs >>> >>> //Aaron >>> >>> >>> >> >
