On 12/5/2017 6:02 AM, Rick Dig wrote: > is it normal to have many instances (100+) of SolrIndexSearchers to be open > at the same time? Our Heap Analysis shows this to be the case. > > We have autoCommit for every 5 minutes, with openSearcher=true, would this > close the old searcher and create a new one or just create a new one with > the old one still not getting dereferenced? if so, when do the older > searchers get cleaned up ?
How many cores is that Solr instance hosting? Also, have you verified that those searcher objects are actually live and not slated for garbage collection? If not, then what I'm saying below may not apply, and you should find out how many are live before going any further. I can imagine a situation where a heap might have over 100 searcher objects, but most of them are actually dead and ready for GC. Each core will have at least one active searcher object at all times. When a commit is made that opens a new searcher, you'll temporarily have an extra searcher object on that core. The old one SHOULD be removed as soon as everything using it (which would include queries) is done with it and the reference counter is fully decremented. This is why Erick asked about custom code. Sometimes when there are Solr plugins that use the searcher object, the author doesn't close them properly, and that can lead to the objects building up and never getting removed. If there are a large number of very slow queries at the same time as updates are happening and there are frequent commits that open a new searcher, then you can end up in a situation where there are several searcher objects which are all running the slow queries at the same time. I'm not going to claim that a bug in Solr where searchers don't get closed is impossible, but we haven't seen any evidence in current versions that this is the case. Resource leaks have happened in Solr, but they're very rare. When I check the list history for the past three years, I only see one other thread you've been involved in, about a month ago. The problems you described at that time and the problem with lots of searcher objects *might* be related to each other, but I can't be sure. Can you put a solr.log file on a sharing/paste website so we can see if there's anything unusual in it? You've said that your autoCommit interval is five minutes, but I suspect that you may have explicit commits happening on a much more frequent basis. The solr.log file would reveal what's actually happening. Do you have autoSoftCommit configured? If so, what is the config? Are you including a commitWithin parameter on your indexing requests? Is there ever any time when the Solr instance is mostly idle, when there are no updates happening and the query rate is zero or very low? If so, can you see how many searchers are found on the heap at that time? Make sure that the heap dump only includes live objects, so objects that can be collected as garbage are not included. Thanks, Shawn
