Ah! Yes - that makes much more sense:
CPU: http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_CPU.jpg
Mem: http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_Mem.jpg
-Joe
On 8/18/2017 3:35 PM, Michael Braun wrote:
When I recommended JVisualVM, specifically the "Sampling" portion of
the app - just using it to see the monitoring isn't half as useful.
On Fri, Aug 18, 2017 at 3:31 PM, Joe Obernberger
<joseph.obernber...@gmail.com <mailto:joseph.obernber...@gmail.com>>
wrote:
Hi Walter - I see what you are saying, but the machine is not
actively swapping (that would be the concern - right?) It's the
CPU usage that I'm trying to figure out. Htop reports that there
is about 20G of disk cache in use, and about 76G of RAM in use by
programs. VIRT memory is what was requested to be allocated, not
what was actually allocated; that is RES. Unless my understanding
of top is wrong.
http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_htop.jpg
<http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_htop.jpg>
atop:
http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_atop.jpg
<http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_atop.jpg>
-Joe
On 8/18/2017 3:12 PM, Walter Underwood wrote:
I see a server with 100Gb of memory and processes (java and
jsvc) using 203Gb of virtual memory. Hmm.
wunder
Walter Underwood
wun...@wunderwood.org <mailto:wun...@wunderwood.org>
http://observer.wunderwood.org/
<http://observer.wunderwood.org/> (my blog)
On Aug 18, 2017, at 12:05 PM, Joe Obernberger
<joseph.obernber...@gmail.com
<mailto:joseph.obernber...@gmail.com>> wrote:
Thank you Shawn. Please see:
http://www.lovehorsepower.com/Vesta
<http://www.lovehorsepower.com/Vesta>
for screen shots of top
(http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_top.jpg
<http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_top.jpg>)
and several screen shots over various times of jvisualvm.
There is also the GC log and the regular solr.log for one
server (named Vesta). Please note that we are using HDFS
for storage. I love top, but also use htop and atop as
they show additional information. In general we are RAM
limited and therefore do not have much cache for OS/disk
as we would like, but this issue is CPU related. After
restarting the one node, the CPU usage stayed low for a
while, but then eventually comes up to ~800% where it will
stay.
Please let me know if there is other information that I
can provide, or what I should be looking for in the GC
logs. Thanks!
-Joe
On 8/18/2017 2:25 PM, Shawn Heisey wrote:
On 8/18/2017 10:37 AM, Joe Obernberger wrote:
Indexing about 15 million documents per day across
100 shards on 45
servers. Up until about 350 million documents,
each of the solr
instances was taking up about 1 core (100% CPU).
Recently, they all
jumped to 700%. Is this normal? Anything that I
can check for?
I don't see anything unusual in the solr logs.
Sample from the GC logs:
A sample from GC logs won't reveal anything. We would
need the entire
GC log. To share something like that, you need a file
sharing site,
something like dropbox. With the full log, we can
analyze it for
indications of GC problems.
There are many things that can cause a sudden massive
increase in CPU
usage. In this case, it is likely due to increased
requirements because
indexing 15 million documents per day has made the
index larger, and now
it probably needs additional resources on each server
that are not
available.
The most common need for additional resources is
unallocated system
memory for the operating system to cache the index.
Something else that
sometimes happens is that the index outgrows the max
heap size, which we
would be able to learn from the full GC log.
These problem are discussed here:
https://wiki.apache.org/solr/SolrPerformanceProblems
<https://wiki.apache.org/solr/SolrPerformanceProblems>
Another useful piece of information is obtained by
running the "top"
utility on the commandline, pressing shift-M to sort
by memory, and
taking a screenshot of that display. Then you would
need a file-sharing
website to share the image.
Thanks,
Shawn
---
This email has been checked for viruses by AVG.
http://www.avg.com
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
Virus-free. www.avg.com
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
