Thanks for your response *Shawn*. Have responded your queries and shared the SOLR details here.
What precisely were you measuring during the night with no activity ? · Earlier we have enabled delta content indexing (content which got updated on day to day basis) script which triggers every 30 minutes. We monitored the load on SOLR when the delta indexing is performed. · CPU utilization by SOLR process, · Heap memory usage. what precise methods were you using to measure it? · We used the “*top*” command to monitor the solr process, · We monitored the free space in the server during the delta change indexing process, · We also enabled JMX remote monitoring in SOLR and used *jConsole* & *jVisualVM* to monitor CPU, Heap & Thread utilizations. What part of the information you obtained represents a problem in your mind? · Information obtained from top and the SOLR GC log in the *linux-a* server · Allocated max heap size is 2 GB. Based on the jConsole monitor h*eap usage is < 600 MB in linux-a server but the CPU utilization by SOLR process is on top (in figures it consumes 0.5 to 5 %)*. There is one solr instance per server i.e in linux-a one solr instance and in linux-b one solr instance (stand alone) is available. linux-a linux-b Number of Cores 49 14 Number of Documents 6145495 (~6M) 1923181 (~1M) Overall index directory size 9.9 GB 14 GB Min Heap Memory 256 MB Max Heap Memory 2048 MB Total # System Processors 40 Overall RAM size 125 GB Java Version 1.8.0_65 64-Bit Server VM SOLR Parameters -Xms512m -Xmx2048m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/app/solr/logs/solr_gc.log -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=17010 -Dcom.sun.management.jmxremote.rmi.port=17010 -Djetty.port=7010 -DSTOP.PORT=6010 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/opt/solr/server -Dsolr.solr.home=/local/solr/default/data -Dsolr.install.dir=/opt/solr -Dlog4j.configuration=file:/app/solr/log4j.properties -Xss256k Regards, Thirukumaran M On Sun, Aug 28, 2016 at 2:01 AM, Shawn Heisey <apa...@elyograg.org> wrote: > On 8/27/2016 9:08 AM, Thiru M wrote: > > We are using Solr 5.4.0 - "stand-alone" mode in our production boxes > > hosted in Red Hat Enterprise Linux (RHEL) OS. > > > > Each box have number of different cores. Have attached the screenshot > > shot with the Solr core & system details. > > > > 1. Earlier indexing was performed every 30 minutes in both production > > servers, > > > > 2. In linux-a server 30 (stand-alone) cores created on same day and > > content indexed into it, > > > > 3. we then spotted unusual GC performing every 2 to 7 seconds in > > linux-a server and the Solr process spiked, > > > > 4. Then we removed the indexing in linux-a server for a week, > > monitored the both Solr process and GC.(No indexing performed during > > this time), > > > > 5. No one uses the system during night time which we ensured it from > > our end. But both Solr process and GC were in its peak, even "during > > non business hours", > > > > 6. Have restated Solr instance in linux-a server. GC started again > > after Solr instance brought up, > > > > 7. In linux-b server no spike in Solr process and no issues with GC, > > > > Indexing creates a LOT of garbage. Queries also create garbage, but not > nearly as fast as indexing. Solr has some background processes, and > these will create garbage too. Java uses a garbage collection memory > model, so this is completely normal for java applications. > > What precisely were you measuring during the night with no activity, and > what precise methods were you using to measure it? What part of the > information you obtained represents a problem in your mind? > > We'll also need some details about these servers: > > * Total index size of all Solr cores on the server. > * Total amount of memory installed in the server. > * Total number of documents contained in all Solr cores. > * How many Solr instances per server? What is the max heap size of each > instance? > > Attachments rarely make it to the list. The screenshot you mentioned > did not make it. You'll need to put it somewhere on the Internet an > provide a URL. Sharing sites like drobox or imgur are good choices for > image data. > > Since I don't have a clear idea of what the exact issue is here, I don't > have any immediate suggestions, aside from possibly increasing your max > heap size ... but depending on the answers to the questions above, that > might make things worse. > > Thanks, > Shawn > > -- By M.Thirukumaran
