What changed in the system? Has there been a code change, increased QPS or different types of queries being run?
Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Jul 7, 2017 at 8:07 AM, Markus Jelsma <markus.jel...@openindex.io> wrote: > Hello, > > No, 800 MB heap is fine. Up till now, heap consumption averages at 400 MB > for past few months, which is fine. FilterCache and QueryCache are always > full. If restarted, they fill up very quickly. FilterCache is just 512 > entries large, QueryCache about 2k. > > All Solr instances run on separate machines (VMs). > > Thanks, > Markus > > > > -----Original message----- > > From:Susheel Kumar <susheel2...@gmail.com> > > Sent: Friday 7th July 2017 13:27 > > To: solr-user@lucene.apache.org > > Subject: Re: Slowly running OOM due to Query instances?! > > > > Xms 800m sounds low regardless did you know how much total cache > > consumption may go based on your current solrconfig.xml settings. Also 2 > > shards and 3 replca's are on 6 such machines i assume. > > > > Thanks, > > Susheel > > > > On Fri, Jul 7, 2017 at 7:01 AM, Markus Jelsma < > markus.jel...@openindex.io> > > wrote: > > > > > Hello, > > > > > > This morning i spotted our QTime suddenly go up. This has been going on > > > for a few hours by now and coincides with a serious increase in heap > > > consumption. No node ran out of memory so far but either that is going > to > > > happen soon, or the nodes become unusable in another manner. > > > > > > I restarted one of the Solr instances and launched VisualVM at it, and > > > some other nodes that use to much heap. Starting the memory sampler, > > > something was obvious straight away. > > > > > > The nodes consuming too much heap all have a serious amount of *Query, > and > > > BooleanClause instances, PayloadScoreQuery, TermQuery, BoostQuery, > > > BooleanQuery, SpanTermQuery and so forth. Lots of Builder and Term > > > instances too, very distinct from the node that was just freshly > restarted. > > > > > > Another peculiarity, some nodes have exactly 65536 instances of > TermQuery > > > and/or BoostQuery, probably unrelated but not something i would have > > > expected to see anyway. > > > > > > So, what's up? We do have a custom query parser extending > EdismaxQParser, > > > it transliterates dates and creates payload and span queries. I may be > > > doing something wrong but i don't know, i have made and used a variety > of > > > QParsers, for many years but this is new. Any hints on where to look, > what > > > to watch out for? > > > > > > Many thanks! > > > Markus > > > > > > Xmx 800m, 8 GB RAM, SSD > > > 2 shards, three replica's > > > replica size ~17 GB, 2.2 million docs/replica > > > > > >
