@ Walter, the daily optimization was introduced as we saw a decrease in the performance for searches that happen during the peak hours - when loads of updates take place on index. The load testing was proved slightly successfull on optimized indexes. As a matter of fact, the merge factor was increased from 10 to 30 to make it acceptable.
@Upayavira , thanks for the inputs. I will try to avoid the daily optimizations however its sort of the workplace policy not to alter anything except the essential configs for this release of project. I take your point that the daily optimizations are unnecessary even then its hard to imagine why they take 6-8 hours a day when previously they were finished within half an hour. @Michael, thank for poitning that out, I will try using solr.NIOFSDirectoryFactory as currently I'm using the default one. Regarding your questions, - Nothing has changed between solr 1.4 and solr 4 except the solr config. I have built 2 separate environments using solr 1.4 and solr 4 with the same application code, db config etc. and can see the difference in the optimization timings. - I will check the solr stats for gc and also during optimization. I see that the index size reaches to 17 Gig from 8.5G and the CPU utilization then is the highest.. And I meant WAS only as in Websphere Application Server. @Otis, a quick google for optimize wunder Erick Otis results in this mail chain (ha ha !), but I will dig the mail archives, thank you for your suggestion.. Have a good day all, I will come back with my findings.. Best, Sandeep On 5 December 2012 06:07, Walter Underwood <wun...@wunderwood.org> wrote: > It was not necessary under 1.4. It has never been necessary. > > It was not necessary in Ultraseek Server in 1996, using the same merging > model. > > In some cases, it can be a good idea. Since you are continuously updating, > this is not one of those cases. > > wunder > > On Dec 4, 2012, at 9:29 PM, Upayavira wrote: > > > I tried that search, without success :-( > > > > I suspect what Otis was trying to say was to question why you are > > optimising. Optimise was necessary under 1.4, but with newer Solr, the > > new TieredMergePolicy does a much better job of handling background > > merging, reducing the need for optimize. Try just not doing it at all > > and see if your index actually reaches a point where it is needed. > > > > Upayavira > > > > On Wed, Dec 5, 2012, at 12:31 AM, Otis Gospodnetic wrote: > >> Hi, > >> > >> You should search the ML archives for : optimize wunder Erick Otis :) > >> > >> Is WAS really AWS? If so, if these are new EC2 instances you are > >> unfortunately unable to do a fair apples to apples comparison. Have you > >> tried a different set of instances? > >> > >> Otis > >> -- > >> Performance Monitoring - http://sematext.com/spm > >> On Dec 4, 2012 6:29 PM, "Sandeep Mestry" <sanmes...@gmail.com> wrote: > >> > >>> Hi All, > >>> > >>> I have recently migrated from solr 1.4 to solr 4 and have done the > basic > >>> changes required for solr 4 in solrconfig.xml and schema.xml. I have > also > >>> rebuilt the index set for solr 4. > >>> We run optimize every morning at 4 am and we keep the index updates off > >>> during this process. > >>> Previously, with 1.4 - the optimization used to take around 20-30 mins > per > >>> shard but now with solr 4, its taking 6-8 hours or even more.. > >>> I have also tested the optimize from solr UI and that takes 6-8 hours > too.. > >>> The hardware is saeme and, we have deployed solr under WAS. > >>> There ar 4 shards and every shard contains around 8 - 9 Gig of data > and we > >>> are using master-slave configuration with rsync. I have not enabled > soft > >>> commit. Also, commiter process is scheduled to run every minute. > >>> > >>> I am not sure which part I'm missing, do let me know your inputs > please. > >>> > >>> Many Thanks in advance, > >>> Sandeep > >>> > > -- > Walter Underwood > wun...@wunderwood.org > > > >
