Reverted back to 7.6.0 - same settings, but now I do not encounter the
large CPU usage.
-Joe
On 2/12/2019 12:37 PM, Joe Obernberger wrote:
Thank you Shawn. Yes, I used the settings off of your site. I've
restarted the cluster and the CPU usage is back up again. Looking at
it now, it doesn't appear to be GC related.
Full log from one of the nodes that is pegging 13 CPU cores:
http://lovehorsepower.com/solr_gc.log.0.current
Thank you For the gceasy.io site - that is very slick! I'll use that
in the future. I can try using the standard settings, but again - at
this point it doesn't look GC related to me?
-Joe
On 2/12/2019 11:35 AM, Shawn Heisey wrote:
On 2/12/2019 7:35 AM, Joe Obernberger wrote:
Yesterday, we upgraded our 40 node cluster from solr 7.6.0 to solr
7.7.0. This morning, all the nodes are using 1200+% of CPU. It
looks like it's in garbage collection. We did reduce our HDFS cache
size from 11G to 6G, but other than that, no other parameters were
changes.
Your message included a small excerpt from the GC log. That is not
helpful. We will need the entire GC log, possibly more than one log.
The log or logs should fully cover the timeframe where the problem
occurs. Full disclosure: Once obtained, I would use this website to
analyze GC log data:
http://gceasy.io
Parameters are:
GC_TUNE="-XX:+UseG1GC \
-XX:MaxDirectMemorySize=6g \
-XX:+PerfDisableSharedMem \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=16m \
-XX:MaxGCPauseMillis=300 \
-XX:InitiatingHeapOccupancyPercent=75 \
-XX:+UseLargePages \
-XX:ParallelGCThreads=16 \
-XX:-ResizePLAB \
-XX:+AggressiveOpts"
Looks like you've chosen to use G1 settings very similar to what I
put on my wiki page:
https://wiki.apache.org/solr/ShawnHeisey#Current_experiments
Those settings are not intended to be a canonical resource that
everyone can use. Your heap size is different than what I was using
when I worked on that, so you may need different settings.
Have you considered not using your own GC tuning, letting Solr's
start script handle that?
With the limited information available, my initial guess is that you
need a larger heap, that Java is spending all its time freeing up
enough memory to keep the program running.
Thanks,
Shawn
---
This email has been checked for viruses by AVG.
https://www.avg.com
