bq. The only custom stuff left is an extension of SearchHandler that only writes numFound to the response headers.
Well, one more to go ;). It's incredibly easy to overlook innocent-seeming calls that increment the underlying reference count of some objects but don't decrement them, usually through a close call. Which isn't necessarily a close if the underlying reference count is still > 0. You may infer that I've been there and done that ;). Sometime the compiler warnings about "resource leak" can help pinpoint those too. Best, Erick On Fri, Jun 29, 2018 at 9:16 AM, Markus Jelsma <markus.jel...@openindex.io> wrote: > Hello Yonik, > > I took one node of the 7.2.1 cluster out of the load balancer so it would > only receive shard queries, this way i could kind of 'safely' disable our > custom components one by one, while keeping functionality in place by letting > the other 7.2.1 nodes continue on with the full configuration. > > I am now at a point where literally all custom components are deleted or > commented out in the config for the node running 7.4. The only custom stuff > left is an extension of SearchHandler that only writes numFound to the > response headers, and all the token filters in our schema. > > You were right, it was leaking exactly one SolrIndexSearcher instance on each > commit. But, with all our stuff gone, the leak is still there! I triple > checked it! Of course, the bastard is locally still not reproducible. > > So, what is next? I have no clues left. > > Many, many thanks, > Markus > > -----Original message----- >> From:Markus Jelsma <markus.jel...@openindex.io> >> Sent: Thursday 28th June 2018 23:52 >> To: solr-user@lucene.apache.org >> Subject: RE: 7.3 appears to leak >> >> Hello Yonik, >> >> If leaking a whole SolrIndexSearcher would cause this problem, then the only >> custom component would be our copy/paste-and-enhance version of the elevator >> component, is the root of all problems. It is a direct copy of the 7.2 >> source where only things like getAnalyzedQuery, the ElevationObj and the >> loop over the map entries is changed. >> >> There are no changes to code related to the searcher. Other component where >> we get a RefCount of searcher is used without issues, we always decrement >> the reference after using it. But those components are not in use in this >> collection. >> >> The source has changed a lot with 7.4 but we still use the old code. I will >> investigate the component thoroughly, even revert to the old 7.2 vanilla >> component for a brief period in production for one machine. It may not be a >> problem if i don't let our load balancer access it directly, so it only >> serves shard queries. >> >> I will get back to this topic tomorrow! >> >> Many thanks, >> Markus >> >> >> >> -----Original message----- >> > From:Yonik Seeley <ysee...@gmail.com> >> > Sent: Thursday 28th June 2018 23:30 >> > To: solr-user@lucene.apache.org >> > Subject: Re: 7.3 appears to leak >> > >> > > * SortedIntDocSet instances ánd ConcurrentLRUCache$CacheEntry instances >> > > are both leaked on commit; >> > >> > If these are actually filterCache entries being leaked, it stands to >> > reason that a whole searcher is being leaked somewhere. >> > >> > -Yonik >> > >>
