You're doing more commits than you need. You may want to turn off autocommit since you are running commit yourself. Every commit causes segment activity, so if you want to minimize that, you don't need autocommit.
About memory sizing: you should drop the memory assigned to Solr until it slows down, then increase it a little. All of the rest should be used by the OS for disk caching. With this much ram, investigate "Large Pages" support. This is an operating system hack to make large programs run faster in large ram machines. On Sat, Apr 14, 2012 at 7:33 PM, Rohit <ro...@in-rev.com> wrote: > Hey Shawn, > > Solr is working better, though not out of the woods, freed up some memory is > the system and also increased the mergeFactor to 20. > > Has another question, we had autocommit ON all this while in our > solrconfig.xml, but since the upgrade we have been noticing keeping > autocommit on is increasing the commit time, though I cannot find a reason > are they related in anyway? > > > Regards, > Rohit > Mobile: +91-9901768202 > About Me: http://about.me/rohitg > > > -----Original Message----- > From: Shawn Heisey [mailto:s...@elyograg.org] > Sent: 13 April 2012 11:01 > To: solr-user@lucene.apache.org > Subject: Re: solr 3.5 taking long to index > > On 4/12/2012 8:42 PM, Rohit wrote: >> The machine has a total ram of around 46GB. My Biggest concern is Solr index >> time gradually increasing and then the commit stops because of timeouts, out >> commit rate is very high, but I am not able to find the root cause of the >> issue. > > For good performance, Solr relies on the OS having enough free RAM to keep > critical portions of the index in the disk cache. Some numbers that I have > collected from your information so far are listed below. > Please let me know if I've got any of this wrong: > > 46GB total RAM > 36GB RAM allocated to Solr > 300GB total index size > > This leaves only 10GB of RAM free to cache 300GB of index, assuming that this > server is dedicated to Solr. The critical portions of your index are very > likely considerably larger than 10GB, which causes constant reading from the > disk for queries and updates. With a high commit rate and a relatively low > mergeFactor of 10, your index will be doing a lot of merging during updates, > and some of those merges are likely to be quite large, further complicating > the I/O situation. > > Another thing that can lead to increasing index update times is cache > warming, also greatly affected by high I/O levels. If you visit the > /solr/corename/admin/stats.jsp#cache URL, you can see the warmupTime for each > cache in milliseconds. > > Adding more memory to the server would probably help things. You'll want to > carefully check all the server and Solr statistics you can to make sure that > memory is the root of problem, before you actually spend the money. At the > server level, look for things like a high iowait CPU percentage. For Solr, > you can turn the logging level up to INFO in the admin interface as well as > turn on the infostream in solrconfig.xml for extensive debugging. > > I hope this is helpful. If not, I can try to come up with more specific > things you can look at. > > Thanks, > Shawn > > -- Lance Norskog goks...@gmail.com
