First start by lengthening your soft and hard commit intervals substantially. Start with 60000 and work backwards I'd say.
Ramkumar has tuned the heck out of his installation to get the commit intervals to be that short ;). I'm betting that you'll see your RAM usage go way down, but that' s a guess until you test. Best, Erick On Sun, Mar 15, 2015 at 10:56 PM, Nitin Solanki <nitinml...@gmail.com> wrote: > Hi Erick, > You are saying correct. Something, **"overlapping searchers" > warning messages** are coming in logs. > **numDocs numbers** are changing when documents are adding at the time of > indexing. > Any help? > > On Sat, Mar 14, 2015 at 11:24 PM, Erick Erickson <erickerick...@gmail.com> > wrote: > >> First, the soft commit interval is very short. Very, very, very, very >> short. 300ms is >> just short of insane unless it's a typo ;). >> >> Here's a long background: >> >> https://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ >> >> But the short form is that you're opening searchers every 300 ms. The >> hard commit is better, >> but every 3 seconds is still far too short IMO. I'd start with soft >> commits of 60000 and hard >> commits of 60000 (60 seconds), meaning that you're going to have to >> wait 1 minute for >> docs to show up unless you explicitly commit. >> >> You're throwing away all the caches configured in solrconfig.xml more >> than 3 times a second, >> executing autowarming, etc, etc, etc.... >> >> Changing these to longer intervals might cure the problem, but if not >> then, as Hoss would >> say, "details matter". I suspect you're also seeing "overlapping >> searchers" warning messages >> in your log, and it;s _possible_ that what's happening is that you're >> just exceeding the >> max warming searchers and never opening a new searcher with the >> newly-indexed documents. >> But that's a total shot in the dark. >> >> How are you looking for docs (and not finding them)? Does the numDocs >> number in >> the solr admin screen change? >> >> >> Best, >> Erick >> >> On Thu, Mar 12, 2015 at 10:27 PM, Nitin Solanki <nitinml...@gmail.com> >> wrote: >> > Hi Alexandre, >> > >> > >> > *Hard Commit* is : >> > >> > <autoCommit> >> > <maxTime>${solr.autoCommit.maxTime:3000}</maxTime> >> > <openSearcher>false</openSearcher> >> > </autoCommit> >> > >> > *Soft Commit* is : >> > >> > <autoSoftCommit> >> > <maxTime>${solr.autoSoftCommit.maxTime:300}</maxTime> >> > </autoSoftCommit> >> > >> > And I am committing 20000 documents each time. >> > Is it good config for committing? >> > Or I am good something wrong ? >> > >> > >> > On Fri, Mar 13, 2015 at 8:52 AM, Alexandre Rafalovitch < >> arafa...@gmail.com> >> > wrote: >> > >> >> What's your commit strategy? Explicit commits? Soft commits/hard >> >> commits (in solrconfig.xml)? >> >> >> >> Regards, >> >> Alex. >> >> ---- >> >> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: >> >> http://www.solr-start.com/ >> >> >> >> >> >> On 12 March 2015 at 23:19, Nitin Solanki <nitinml...@gmail.com> wrote: >> >> > Hello, >> >> > I have written a python script to do 20000 documents >> indexing >> >> > each time on Solr. I have 28 GB RAM with 8 CPU. >> >> > When I started indexing, at that time 15 GB RAM was freed. While >> >> indexing, >> >> > all RAM is consumed but **not** a single document is indexed. Why so? >> >> > And it through *HTTPError: HTTP Error 503: Service Unavailable* in >> python >> >> > script. >> >> > I think it is due to heavy load on Zookeeper by which all nodes went >> >> down. >> >> > I am not sure about that. Any help please.. >> >> > Or anything else is happening.. >> >> > And how to overcome this issue. >> >> > Please assist me towards right path. >> >> > Thanks.. >> >> > >> >> > Warm Regards, >> >> > Nitin Solanki >> >> >>
