Hi, Emir, My system used to run with max 32GB, the response time is bad as well. swap is set to 4GB, there 3.2 free, I doubt swap would affect it since there is such huge free memory.
I could try to with set Xms and Xmx to the same value, but I doubt how much would that change the response time. BRs //Aaron ________________________________ From: Emir Arnautović <emir.arnauto...@sematext.com> Sent: Monday, March 18, 2019 2:19:19 PM To: solr-user@lucene.apache.org Subject: Re: Solr index slow response Hi Aaron, Without looking too much into numbers, my bet would be that it is large heap that is causing issues. I would decrease is significantly (<30GB) and see if it is enough for your max load. Also, disable swap or reduce swappiness to min. In any case, you should install some monitoring tool that would help you do better analysis when you run into problems. One such tool is our monitoring solution: https://sematext.com/spm HTH, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 18 Mar 2019, at 13:14, Aaron Yingcai Sun <y...@vizrt.com> wrote: > > Hello, Emir, > > Thanks for the reply, this is the solr version and heap info, standalone > single solr server. I don't have monitor tool connected. only look at 'top', > has not seen cpu spike so far, when the slow response happens, cpu usage is > not high at all, around 30%. > > > # curl 'http://.../solr/admin/info/system?wt=json&indent=true' > { > "responseHeader":{ > "status":0, > "QTime":27}, > "mode":"std", > "solr_home":"/ardome/solr", > "lucene":{ > "solr-spec-version":"6.5.1", > "solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - > jimczi - 2017-04-21 12:23:42", > "lucene-spec-version":"6.5.1", > "lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - > jimczi - 2017-04-21 12:17:15"}, > "jvm":{ > "version":"1.8.0_144 25.144-b01", > "name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM", > "spec":{ > "vendor":"Oracle Corporation", > "name":"Java Platform API Specification", > "version":"1.8"}, > "jre":{ > "vendor":"Oracle Corporation", > "version":"1.8.0_144"}, > "vm":{ > "vendor":"Oracle Corporation", > "name":"Java HotSpot(TM) 64-Bit Server VM", > "version":"25.144-b01"}, > "processors":32, > "memory":{ > "free":"69.1 GB", > "total":"180.2 GB", > "max":"266.7 GB", > "used":"111 GB (%41.6)", > "raw":{ > "free":74238728336, > "total":193470136320, > "max":286331502592, > "used":119231407984, > "used%":41.64103736566334}}, > "jmx":{ > > "bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes", > "classpath":"...", > "commandLineArgs":["-Xms100G", > "-Xmx300G", > "-DSTOP.PORT=8079", > "-DSTOP.KEY=..", > "-Dsolr.solr.home=..", > "-Djetty.port=8983"], > "startTime":"2019-03-18T09:35:27.892Z", > "upTimeMS":9258422}}, > "system":{ > "name":"Linux", > "arch":"amd64", > "availableProcessors":32, > "systemLoadAverage":14.72, > "version":"3.0.101-311.g08a8a9d-default", > "committedVirtualMemorySize":2547960700928, > "freePhysicalMemorySize":4530696192, > "freeSwapSpaceSize":3486846976, > "processCpuLoad":0.3257436126790475, > "processCpuTime":93869450000000, > "systemCpuLoad":0.3279781055816521, > "totalPhysicalMemorySize":406480175104, > "totalSwapSpaceSize":4302303232, > "maxFileDescriptorCount":32768, > "openFileDescriptorCount":385, > "uname":"Linux ... 3.0.101-311.g08a8a9d-default #1 SMP Wed Dec 14 10:15:37 > UTC 2016 (08a8a9d) x86_64 x86_64 x86_64 GNU/Linux\n", > "uptime":" 13:09pm up 5 days 21:23, 7 users, load average: 14.72, > 12.28, 11.48\n"}} > > > > > ________________________________ > From: Emir Arnautović <emir.arnauto...@sematext.com> > Sent: Monday, March 18, 2019 12:10:30 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr index slow response > > Hi Aaron, > Which version of Solr? How did you configure your heap? Is it standalone Solr > or SolrCloud? A single server? Do you use some monitoring tool? Do you see > some spikes, pauses or CPU usage is constant? > > Thanks, > Emir > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > >> On 18 Mar 2019, at 11:47, Aaron Yingcai Sun <y...@vizrt.com> wrote: >> >> Hello, Solr! >> >> >> We are having some performance issue when try to send documents for solr to >> index. The repose time is very slow and unpredictable some time. >> >> >> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, while >> 300 GB is reserved for solr, while this happening, cpu usage is around 30%, >> mem usage is 34%. io also look ok according to iotop. SSD disk. >> >> >> Our application send 100 documents to solr per request, json encoded. the >> size is around 5M each time. some times the response time is under 1 >> seconds, some times could be 300 seconds, the slow response happens very >> often. >> >> >> "Soft AutoCommit: disabled", "Hard AutoCommit: if uncommited for 3600000ms; >> if 1000000 uncommited docs" >> >> >> There are around 100 clients sending those documents at the same time, but >> each for the client is blocking call which wait the http response then send >> the next one. >> >> >> I tried to make the number of documents smaller in one request, such as 20, >> but still I see slow response time to time, like 80 seconds. >> >> >> Would you help to give some hint how improve the response time? solr does >> not seems very loaded, there must be a way to make the response faster. >> >> >> BRs >> >> //Aaron >> >> >> >
