Thanks very much for the useful info Eric. Sincerely appreciate you pointed out those questions. In fact, I am currently working with a third party product called Sitecore Web Content Management System (WCMS) that does the job to issue updates to the Solr Index. I need to understand abit more when and how they are committing documents to Solr. Your question about how the new searchers arise to produce those performance warning messages is valid.
Your questions will be more for the Sitecore Product Team to investigate in which I will chase them up for answers. 1. Whether we are overriding the system variable solr.autoSoftCommit.maxTime? 2. Whether we are overriding the solr.autoCommit.maxTime (although this really shouldn't matter). 3. Why we have too many hard commits? 4. How come new searchers are opened even when this is set with <openSearcher>false</openSearcher> I shall chase these questions up with them. Will feedback to you where necessary. Thanks. Best regards, Adrian Liew -----Original Message----- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Wednesday, August 12, 2015 11:19 PM To: solr-user@lucene.apache.org Subject: Re: Performance warning overlapping onDeckSearchers Something's not adding up here. Is your _client_ perhaps issuing commits when you index documents? This is Joel's question, so we need to see how you send docs to Solr. We really need to know how you're indexing docs to Solr. My bet (and I suspect Joel's) is that you're either using SolrJ to send docs to Solr and have something like ****solrJ while (more docs) { create a doc send it to Solr commit } rather than while (more docs) { create a bunch of docs (I usually start with 1,000) using the "commitwithin" option and make it as long as possible send the batch to solr } Maybe commit at the very end, but that's not necessary if you're willing to wait for "commitWithin". ****post.jar Or, you're using post.jar in some kind of loop which commits every time you use it by default. You can disable this, try 'java -jar post.jar -help' for all the options, but the one you want is -Dcommit=no. NOTE: you have to issue a commit _sometime_ to see the docs, either the commitWithin option in SolrJ or explicitly if you're using the post.jar tool. You can even issue a commit (this is suitable for testing) via curl or a browser with http://solr_node:8983/solr/collection/update?commit=true The reason we're focusing here is that: Soft commits are disabled in your setup, this is the "-1" in autoSoftCommit. Hard commits are not opening searchers, this is the autoCommit, <openSearcher>false</openSearcher> section. Are, you perhaps overriding the system variable solr.autoSoftCommit.maxTime when you start up Solr? What about solr.autoCommit.maxTime (although this really shouldn't matter). If you're not overriding the above, then no searchers should be being opened at all after you start Solr, and only one should be opening when you do start Solr. So you should not be getting the warning about Overlapping onDeckSearchers. Forget the static warming queries, they are irrelevant until we understand why you're getting any new searchers. For future reference, these are the newSearcher and firstSearcher events in solrconfig.xml. newSearcher is fired every time one commits, firstSearcher when you start Solr. The bottom line here is you need to find out why you're committing at all, which opens a new searcher which, when that happens too often generates the warning you're seeing. Best, Erick On Wed, Aug 12, 2015 at 6:51 AM, Adrian Liew <adrian.l...@avanade.com> wrote: > Hi Joel, > > I am fairly new to Solr (version which I am using v5.2.1) so I suppose what > you may be asking is referring to the autocommits section: > > > <!-- AutoCommit > > Perform a hard commit automatically under certain conditions. > Instead of enabling autoCommit, consider using "commitWithin" > when adding documents. > > http://wiki.apache.org/solr/UpdateXmlMessages > > maxDocs - Maximum number of documents to add since the last > commit before automatically triggering a new commit. > > maxTime - Maximum amount of time in ms that is allowed to pass > since a document was added before automatically > triggering a new commit. > openSearcher - if false, the commit causes recent index changes > to be flushed to stable storage, but does not cause a new > searcher to be opened to make those changes visible. > > If the updateLog is enabled, then it's highly recommended to > have some sort of hard autoCommit to limit the log size. > --> > <autoCommit> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> > <openSearcher>false</openSearcher> > </autoCommit> > > <!-- softAutoCommit is like autoCommit except it causes a > 'soft' commit which only ensures that changes are visible > but does not ensure that data is synced to disk. This is > faster and more near-realtime friendly than a hard commit. > --> > > <autoSoftCommit> > <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime> > </autoSoftCommit> > > Currently, this has been left with the defaults. Are you able to determine if > I have autocommits turned on or off? > > What do you mean by explicit commit? Does this mean a regular commit? > > Quick commits are fairly intermittent. Although, we will be running some > automated tasks to import some content which may directly cause frequent > commits. This particular warning does not happen often but does happen > frequently within a one minute period. I predict with automated import of > content, this can trigger the warnings quite frequently. I have seen an error > reported back where the number of maxWarmingSearchers have been exceeded. > > In regards with checking to see if there are any static warm queries, how do > you advise which section to look at in the solrconfig.xml? > > Best regards, > > Adrian Liew > > > -----Original Message----- > From: Joel Bernstein [mailto:joels...@gmail.com] > Sent: Wednesday, August 12, 2015 6:35 PM > To: solr-user@lucene.apache.org > Subject: Re: Performance warning overlapping onDeckSearchers > > You'll want to check to see if there are any static warming queries being run > as well. > > How often are you committing and opening a new searcher? Are you using > autoCommits or explicitly committing? > > Joel Bernstein > http://joelsolr.blogspot.com/ > > On Wed, Aug 12, 2015 at 3:57 AM, Adrian Liew <adrian.l...@avanade.com> > wrote: > >> Additionally, >> >> I realized that my autowarmCount is set to zero for the following >> Cache entries except perSegFilter: >> >> <filterCache class="solr.FastLRUCache" >> size="512" >> initialSize="512" >> autowarmCount="0"/> >> >> <!-- Query Result Cache >> >> Caches results of searches - ordered lists of document ids >> (DocList) based on a query, a sort, and the range of >> documents requested. >> Additional supported parameter by LRUCache: >> maxRamMB - the maximum amount of RAM (in MB) that this >> cache is allowed >> to occupy >> --> >> <queryResultCache class="solr.LRUCache" >> size="512" >> initialSize="512" >> autowarmCount="0"/> >> >> <!-- Document Cache >> >> Caches Lucene Document objects (the stored fields for each >> document). Since Lucene internal document ids are transient, >> this cache will not be autowarmed. >> --> >> <documentCache class="solr.LRUCache" >> size="512" >> initialSize="512" >> autowarmCount="0"/> >> >> <!-- custom cache currently used by block join --> >> <cache name="perSegFilter" >> class="solr.search.LRUCache" >> size="10" >> initialSize="0" >> autowarmCount="10" >> regenerator="solr.NoOpRegenerator" /> >> >> >> The link, >> https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Ov >> e rlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F >> did suggest to reduce the autowarmCount or reduce warm up cache >> activity (which I am not sure where to begin doing this). >> >> I suspect autowarmCount is not very large as the above. >> >> Let me know what you think. >> >> Best regards, >> Adrian Liew >> >> >> -----Original Message----- >> From: Adrian Liew [mailto:adrian.l...@avanade.com] >> Sent: Wednesday, August 12, 2015 3:32 PM >> To: solr-user@lucene.apache.org >> Subject: RE: Performance warning overlapping onDeckSearchers >> >> Thanks Shawn. Having said that increasing maxWarmingSearchers is >> usually wrong to solve this, are there any implications if we set >> maxWarmingSearchers to zero to resolve this problem? >> >> Or do you think there are some other settings that are worthwhile >> tuning to cater to the above? >> >> Best regards, >> Adrian >> >> -----Original Message----- >> From: Shawn Heisey [mailto:apa...@elyograg.org] >> Sent: Tuesday, August 11, 2015 11:02 PM >> To: solr-user@lucene.apache.org >> Subject: Re: Performance warning overlapping onDeckSearchers >> >> On 8/11/2015 3:02 AM, Adrian Liew wrote: >> > Has anyone come across this issue, [some_index] PERFORMANCE WARNING: >> Overlapping onDeckSearchers=2? >> > >> > I am currently using Solr v5.2.1. >> > >> > What does this mean? Does this raise red flags? >> > >> > I am currently encountering an issue whereby my Sitecore system is >> unable to update the index appropriately. I am not sure if this is >> linked to the warnings above. >> >> >> https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Ov >> e rlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F >> >> What the wiki page doesn't explicitly state is that increasing >> maxWarmingSearchers is usually the wrong way to solve this, because >> that can actually make the problem *worse*. It is implied by the >> things the page DOES say, but it is not stated. >> >> Thanks, >> Shawn >> >>
