Hi Walter - I see what you are saying, but the machine is not actively
swapping (that would be the concern - right?) It's the CPU usage that
I'm trying to figure out. Htop reports that there is about 20G of disk
cache in use, and about 76G of RAM in use by programs. VIRT memory is
what was requested to be allocated, not what was actually allocated;
that is RES. Unless my understanding of top is wrong.
http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_htop.jpg
atop:
http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_atop.jpg
-Joe
On 8/18/2017 3:12 PM, Walter Underwood wrote:
I see a server with 100Gb of memory and processes (java and jsvc) using 203Gb
of virtual memory. Hmm.
wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/ (my blog)
On Aug 18, 2017, at 12:05 PM, Joe Obernberger <joseph.obernber...@gmail.com>
wrote:
Thank you Shawn. Please see:
http://www.lovehorsepower.com/Vesta
for screen shots of top
(http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_top.jpg) and several screen
shots over various times of jvisualvm.
There is also the GC log and the regular solr.log for one server (named Vesta).
Please note that we are using HDFS for storage. I love top, but also use htop
and atop as they show additional information. In general we are RAM limited
and therefore do not have much cache for OS/disk as we would like, but this
issue is CPU related. After restarting the one node, the CPU usage stayed low
for a while, but then eventually comes up to ~800% where it will stay.
Please let me know if there is other information that I can provide, or what I
should be looking for in the GC logs. Thanks!
-Joe
On 8/18/2017 2:25 PM, Shawn Heisey wrote:
On 8/18/2017 10:37 AM, Joe Obernberger wrote:
Indexing about 15 million documents per day across 100 shards on 45
servers. Up until about 350 million documents, each of the solr
instances was taking up about 1 core (100% CPU). Recently, they all
jumped to 700%. Is this normal? Anything that I can check for?
I don't see anything unusual in the solr logs. Sample from the GC logs:
A sample from GC logs won't reveal anything. We would need the entire
GC log. To share something like that, you need a file sharing site,
something like dropbox. With the full log, we can analyze it for
indications of GC problems.
There are many things that can cause a sudden massive increase in CPU
usage. In this case, it is likely due to increased requirements because
indexing 15 million documents per day has made the index larger, and now
it probably needs additional resources on each server that are not
available.
The most common need for additional resources is unallocated system
memory for the operating system to cache the index. Something else that
sometimes happens is that the index outgrows the max heap size, which we
would be able to learn from the full GC log.
These problem are discussed here:
https://wiki.apache.org/solr/SolrPerformanceProblems
Another useful piece of information is obtained by running the "top"
utility on the commandline, pressing shift-M to sort by memory, and
taking a screenshot of that display. Then you would need a file-sharing
website to share the image.
Thanks,
Shawn
---
This email has been checked for viruses by AVG.
http://www.avg.com
