Hi Thiru,
Two Solrs with different data and usage patterns should be tuned
separately and comparing one to another does not give much value.
Like Shawn suggested, first thing that you can try is increase heap
size. Having different Xms and Xmx is bad practice so make sure it is
set to the same value. Even you have enough RAM on your machines, start
with smaller heap (e.g. 4GB) and monitor JVM. Heap should be just about
large to handle your top load.
You could also use more advanced monitoring tool that will allow you to
correlate Solr activities with JVM/OS metrics. One such tool is our's
SPM: http://sematext.com/spm.
Regards,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On 31.08.2016 06:28, Thiru M wrote:
Thanks for your response *Shawn*.
Have responded your queries and shared the SOLR details here.
What precisely were you measuring during the night with no activity ?
· Earlier we have enabled delta content indexing (content which got
updated on day to day basis) script which triggers every 30 minutes. We
monitored the load on SOLR when the delta indexing is performed.
· CPU utilization by SOLR process,
· Heap memory usage.
what precise methods were you using to measure it?
· We used the “*top*” command to monitor the solr process,
· We monitored the free space in the server during the delta change
indexing process,
· We also enabled JMX remote monitoring in SOLR and used *jConsole*
& *jVisualVM* to monitor CPU, Heap & Thread utilizations.
What part of the information you obtained represents a problem in your mind?
· Information obtained from top and the SOLR GC log in the *linux-a*
server
· Allocated max heap size is 2 GB. Based on the jConsole monitor h*eap
usage is < 600 MB in linux-a server but the CPU utilization by SOLR process
is on top (in figures it consumes 0.5 to 5 %)*.
There is one solr instance per server i.e in linux-a one solr instance and
in linux-b one solr instance (stand alone) is available.
linux-a
linux-b
Number of Cores
49
14
Number of Documents
6145495 (~6M)
1923181 (~1M)
Overall index directory size
9.9 GB
14 GB
Min Heap Memory
256 MB
Max Heap Memory
2048 MB
Total # System Processors
40
Overall RAM size
125 GB
Java Version
1.8.0_65 64-Bit Server VM
SOLR Parameters
-Xms512m
-Xmx2048m
-XX:NewRatio=3
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:ConcGCThreads=4
-XX:ParallelGCThreads=4
-XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-verbose:gc
-XX:+PrintHeapAtGC
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-Xloggc:/app/solr/logs/solr_gc.log
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=17010
-Dcom.sun.management.jmxremote.rmi.port=17010
-Djetty.port=7010
-DSTOP.PORT=6010
-DSTOP.KEY=solrrocks
-Duser.timezone=UTC
-Djetty.home=/opt/solr/server
-Dsolr.solr.home=/local/solr/default/data
-Dsolr.install.dir=/opt/solr
-Dlog4j.configuration=file:/app/solr/log4j.properties
-Xss256k
Regards,
Thirukumaran M
On Sun, Aug 28, 2016 at 2:01 AM, Shawn Heisey <apa...@elyograg.org> wrote:
On 8/27/2016 9:08 AM, Thiru M wrote:
We are using Solr 5.4.0 - "stand-alone" mode in our production boxes
hosted in Red Hat Enterprise Linux (RHEL) OS.
Each box have number of different cores. Have attached the screenshot
shot with the Solr core & system details.
1. Earlier indexing was performed every 30 minutes in both production
servers,
2. In linux-a server 30 (stand-alone) cores created on same day and
content indexed into it,
3. we then spotted unusual GC performing every 2 to 7 seconds in
linux-a server and the Solr process spiked,
4. Then we removed the indexing in linux-a server for a week,
monitored the both Solr process and GC.(No indexing performed during
this time),
5. No one uses the system during night time which we ensured it from
our end. But both Solr process and GC were in its peak, even "during
non business hours",
6. Have restated Solr instance in linux-a server. GC started again
after Solr instance brought up,
7. In linux-b server no spike in Solr process and no issues with GC,
Indexing creates a LOT of garbage. Queries also create garbage, but not
nearly as fast as indexing. Solr has some background processes, and
these will create garbage too. Java uses a garbage collection memory
model, so this is completely normal for java applications.
What precisely were you measuring during the night with no activity, and
what precise methods were you using to measure it? What part of the
information you obtained represents a problem in your mind?
We'll also need some details about these servers:
* Total index size of all Solr cores on the server.
* Total amount of memory installed in the server.
* Total number of documents contained in all Solr cores.
* How many Solr instances per server? What is the max heap size of each
instance?
Attachments rarely make it to the list. The screenshot you mentioned
did not make it. You'll need to put it somewhere on the Internet an
provide a URL. Sharing sites like drobox or imgur are good choices for
image data.
Since I don't have a clear idea of what the exact issue is here, I don't
have any immediate suggestions, aside from possibly increasing your max
heap size ... but depending on the answers to the questions above, that
might make things worse.
Thanks,
Shawn
