Thank you Shawn. Please see:
http://www.lovehorsepower.com/Vesta
for screen shots of top
(http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_top.jpg) and several
screen shots over various times of jvisualvm.
There is also the GC log and the regular solr.log for one server (named
Vesta). Please note that we are using HDFS for storage. I love top,
but also use htop and atop as they show additional information. In
general we are RAM limited and therefore do not have much cache for
OS/disk as we would like, but this issue is CPU related. After
restarting the one node, the CPU usage stayed low for a while, but then
eventually comes up to ~800% where it will stay.
Please let me know if there is other information that I can provide, or
what I should be looking for in the GC logs. Thanks!
-Joe
On 8/18/2017 2:25 PM, Shawn Heisey wrote:
On 8/18/2017 10:37 AM, Joe Obernberger wrote:
Indexing about 15 million documents per day across 100 shards on 45
servers. Up until about 350 million documents, each of the solr
instances was taking up about 1 core (100% CPU). Recently, they all
jumped to 700%. Is this normal? Anything that I can check for?
I don't see anything unusual in the solr logs. Sample from the GC logs:
A sample from GC logs won't reveal anything. We would need the entire
GC log. To share something like that, you need a file sharing site,
something like dropbox. With the full log, we can analyze it for
indications of GC problems.
There are many things that can cause a sudden massive increase in CPU
usage. In this case, it is likely due to increased requirements because
indexing 15 million documents per day has made the index larger, and now
it probably needs additional resources on each server that are not
available.
The most common need for additional resources is unallocated system
memory for the operating system to cache the index. Something else that
sometimes happens is that the index outgrows the max heap size, which we
would be able to learn from the full GC log.
These problem are discussed here:
https://wiki.apache.org/solr/SolrPerformanceProblems
Another useful piece of information is obtained by running the "top"
utility on the commandline, pressing shift-M to sort by memory, and
taking a screenshot of that display. Then you would need a file-sharing
website to share the image.
Thanks,
Shawn
---
This email has been checked for viruses by AVG.
http://www.avg.com
