Hi Leo, Errors that you are seeing are related to frequent commits - new commits is issued before searcher for previous commit is opened and warmed.
I haven’t looked in indexing code in a while, but if assume that it did not change, commits and writes are mutually exclusive - guarded by the same lock. So yes - frequent commits will result in longer index request latency. Also each commit results in new segment and segments are merged - small segment and merge is fast, but still overhead. I would suggest that you change explicit commits with at least commitWithin=100 (or as much as possible) - that will result in at least 100ms worth of documents being grouped into single commit. Also, if you are committing this frequently, think of turning off your caches since they are invalidated on each commit. HTH, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 13 Oct 2017, at 13:06, Leo Prince <leoprince.discussi...@gmail.com> wrote: > > Hi Emir, > > Thanks for the response. > > We have specific near realtime search requirements, that is why we are > explicitly invoking Solr commits. However we will try to improve on > reducing the commits from application. In the meantime, the errors/warnings > I mentioned in my previous mail; are they really due to frequent commits..? > > We are having plenty of requests taking more seconds to deliver from solr > irrespective of network overhead, so any thoughts whether commit frequency > affects solr latency..? > > Thanks, > Leo Prince > > On Fri, Oct 13, 2017 at 2:46 PM, Emir Arnautović < > emir.arnauto...@sematext.com> wrote: > >> Hi Leo, >> It is considered a bad practice to commit from your application. You >> should let Solr handle commits. There is a great article about soft and >> hard commits: https://lucidworks.com/2013/08/23/understanding- >> transaction-logs-softcommit-and-commit-in-sorlcloud/ < >> https://lucidworks.com/2013/08/23/understanding- >> transaction-logs-softcommit-and-commit-in-sorlcloud/> >> >> If you really want to commit from your application, then you should use >> commitWithin parameter that would group your commits in a single commit, >> and you would be sure that your changes are committed within some time (you >> can do that with autocommit as well). >> >> Opening searcher can be fast but it can also last for a while - it depends >> on warming parameters that you set. In any case, I would not recommend that >> you focus on making opening last less, but to see what are your NRT >> requirements and commit as rare as possible. >> >> HTH, >> Emir >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >> >> >> >>> On 13 Oct 2017, at 10:30, Leo Prince <leoprince.discussi...@gmail.com> >> wrote: >>> >>> Hi, >>> >>> I am new to community and thank you for letting me in. >>> >>> Let me get into my concern real quick. Please find my OS and Solr >> versions >>> >>> Ubuntu 14.04.4 LTS >>> solr-spec 4.10.2 >>> solr-impl 4.10.2 1634293 - mike - 2014-10-26 05:56:21 >>> lucene-spec 4.10.2 >>> lucene-impl 4.10.2 1634293 - mike - 2014-10-26 05:51:56 >>> >>> I am getting the following errors/warnings from Solr >>> >>> 1, ERROR: org.apache.solr.client.solrj.impl.HttpSolrServer$ >> RemoteSolrException: >>> Error opening new searcher. exceeded limit of maxWarmingSearchers=2, >>> try again later. >>> 2, PERFORMANCE WARNING: Overlapping onDeckSearchers=2 >>> 3, WARN: DistributedUpdateProcessor error sending update >>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: >>> Error opening new searcher. exceeded limit of maxWarmingSearchers=2, >>> try again later. >>> >>> I have gone through various discussion threads and was able to get a >>> minor idea on scenarios where maxWarmingSearchers exceeds. We have >>> commit doing from application for individual requests and autocommit >>> enabled. >>> >>> So my concern is, is there any chance of performance issues when >>> number of commits are high at a particular point of time. In our >>> application, we are approximating like 100-500 commits can happen >>> simultaneously from application and autocommit too for those >>> individual requests which are not committing individually after the >>> write. >>> >>> Autocommit is configured as follows, >>> >>> <autoCommit> >>> <maxTime>15000</maxTime> >>> <openSearcher>false</openSearcher> >>> </autoCommit> >>> >>> >>> When worked on the openSearcher parameter, I see it makes solr not to >>> open a new search while commit is undergoing. How many seconds or >>> milliseconds, solr takes to complete a commit and allow further new >>> searches to spawn..? Considering we have a massive numerous individual >>> commits and massive doc autocommit, is setting up openSearcher to >>> false, a good practice..? What happens to the new requests when a >>> commit is going on. Will it be queued for new search to spawn or will >>> it be delivered with the old search existed..? >>> >>> Thanks in advance, >>> >>> Leo Prince. >> >>
