Thanks for the response. We got it resolved! .
We made small indexes in bulk using SOLR with Standard File Format and then
merged it with a Lucene app which for some reason made it CFS. Now when we
started adding real time documents using SOLR (with Compound File Format set
to false) it was merging with every commit!
We just set the CFF to true and now its normal. Weird but that's how it got
resolved.
BTW any idea why this happening and if we now optimize it using SFF it
should be fine in future with CFF= false?
P.S: Increasing the MergeFactor didn't even work.
On Wed, Apr 27, 2011 at 10:09 PM, Burton-West, Tom <tburt...@umich.edu>wrote:
> Hi Salman,
>
> Sounds like somehow you are triggering merges or optimizes. What is your
> mergeFactor?
>
> Have you turned on the IndexWriter log?
>
> In solrconfig.xml
> <infoStream file="${solr.indexwriter.log.dir}">true</infoStream>
>
> In our case we feed the directory name as a Java property in our java
> startup script , but you can also hard code where you want the log written
> like in the current example Solr config:
>
> <infoStream file="INFOSTREAM.txt">false</infoStream>
>
> That should provide some clues. For example you can see how many segments
> of each level there are just before you do the commit that triggers the
> problem. My first guess is that you have enough segments so that adding
> the documents and committing triggers a cascading merge. (But this is a WAG
> without seeing what's in your indexwriter log)
>
> Can you also send your solrconfig so we can see your mergeFactor and
> ramBufferSizeMB settings?
>
> Tom
>
> > > All,
> > >
> > > We have created index with CommonGrams and the final size is around
> > 370GB.
> > > Everything is working fine but now when we add more documents into
> index
> > it
> > > takes forever (almost 12 hours)...seems to change all the segments file
> > in a
> > > commit.
> > >
> > > The same commit used to take few mins with normal index.
> > >
> > > Any idea whats going on?
> > >
> > > --
> > > Regards,
> > >
> > > Salman Akram
> > > Principal Software Engineer - Tech Lead
> > > NorthBay Solutions
> > > 410-G4 Johar Town, Lahore
> > > Off: +92-42-35290152
> > >
> > > Cell: +92-321-4391210 -- +92-300-4009941
> > >
> >
>
>
>
> --
> Regards,
>
> Salman Akram
>
--
Regards,
Salman Akram
