I see a server with 100Gb of memory and processes (java and jsvc) using 203Gb of virtual memory. Hmm.
wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Aug 18, 2017, at 12:05 PM, Joe Obernberger <joseph.obernber...@gmail.com> > wrote: > > Thank you Shawn. Please see: > http://www.lovehorsepower.com/Vesta > for screen shots of top > (http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_top.jpg) and several > screen shots over various times of jvisualvm. > > There is also the GC log and the regular solr.log for one server (named > Vesta). Please note that we are using HDFS for storage. I love top, but > also use htop and atop as they show additional information. In general we > are RAM limited and therefore do not have much cache for OS/disk as we would > like, but this issue is CPU related. After restarting the one node, the CPU > usage stayed low for a while, but then eventually comes up to ~800% where it > will stay. > > Please let me know if there is other information that I can provide, or what > I should be looking for in the GC logs. Thanks! > > -Joe > > > On 8/18/2017 2:25 PM, Shawn Heisey wrote: >> On 8/18/2017 10:37 AM, Joe Obernberger wrote: >>> Indexing about 15 million documents per day across 100 shards on 45 >>> servers. Up until about 350 million documents, each of the solr >>> instances was taking up about 1 core (100% CPU). Recently, they all >>> jumped to 700%. Is this normal? Anything that I can check for? >>> >>> I don't see anything unusual in the solr logs. Sample from the GC logs: >> A sample from GC logs won't reveal anything. We would need the entire >> GC log. To share something like that, you need a file sharing site, >> something like dropbox. With the full log, we can analyze it for >> indications of GC problems. >> >> There are many things that can cause a sudden massive increase in CPU >> usage. In this case, it is likely due to increased requirements because >> indexing 15 million documents per day has made the index larger, and now >> it probably needs additional resources on each server that are not >> available. >> >> The most common need for additional resources is unallocated system >> memory for the operating system to cache the index. Something else that >> sometimes happens is that the index outgrows the max heap size, which we >> would be able to learn from the full GC log. >> >> These problem are discussed here: >> >> https://wiki.apache.org/solr/SolrPerformanceProblems >> >> Another useful piece of information is obtained by running the "top" >> utility on the commandline, pressing shift-M to sort by memory, and >> taking a screenshot of that display. Then you would need a file-sharing >> website to share the image. >> >> Thanks, >> Shawn >> >> --- >> This email has been checked for viruses by AVG. >> http://www.avg.com >> >
