Regarding the large number of files, even after optimize, we found that
when rebuilding a large, experimental 1.7TB index on Solr 3.5, instead of
Solr 1.4.1, there were a ton of index files, thousands, in 3.5, when there
used to be just 10 (or 11?) segments worth (as expected with mergeFactor
set to 10) in 1.4.1.
The apparent cause was a Solr switch to use TieredMergePolicy by default
somewhere in the 3.x version series. TieredMergePolicy has a default
segment size limit of 5GB, so if your index goes over 50GB, a mergeFactor
of 10 effectively gets ignored. We remedied this by explicitly configuring
TieredMergePolicy's segment size (and some other things that may or may
not be making a difference) in solrconfig.xml:
<mainIndex>
....
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<int name="maxMergeAtOnce">10</int>
<int name="segmentsPerTier">10</int>
<double name="maxMergedSegmentMB">30000</double>
</mergePolicy>
</mainIndex>
-- Bryan
> -----Original Message-----
> From: avenka [mailto:ave...@gmail.com]
> Sent: Tuesday, June 26, 2012 8:46 AM
> To: solr-user@lucene.apache.org
> Subject: Re: solr java.lang.NullPointerException on select queries
>
> So, I tried 'optimize', but it failed because of lack of space on the
> first
> machine. I then moved the whole thing to a different machine where the
> index
> was pretty much the only thing and was using about 37% of disk, but it
> still
> failed because of a "No space left on device" IOException. Also, the
size
> of
> the index has since doubled to roughly 74% of the disk on this second
> machine now and the number of files has increased from 3289 to 3329.
> Actually even the 3289 files on the first machine were after I tried
> optimize on it once, so the "original" size must have been even smaller.
>
> I don't think I can afford any more space and am close to giving up and
> reclaiming space on the two machines. A couple more questions before
that:
>
> 1) I am tempted to try editing binary--the "magnetic needle" option.
Could
> you elaborate on this? Would there be a way to go back to an index that
is
> the original size from its super-sized current form(s)?
>
> 2) Will CheckIndex also need more than twice the space? Would there be a
> way
> to bring down the size to the original size without running 'optimize'
if
> I
> try that route? Also how exactly do I run CheckIndex, e.g., the exact
URL
> I
> need to hit?
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/solr-
> java-lang-NullPointerException-on-select-queries-tp3989974p3991400.html
> Sent from the Solr - User mailing list archive at Nabble.com.
