1) Do you use compound files (CFS)? This adds a lot of overhead to merging.
2) Does ES use the same merge policy code as Solr?
In solrconfig.xml, here are the lines that control segment merging. You can
probably set mergeFactor to 20 and cut the amount of disk I/O.
<!-- Expert: Merge Policy
The Merge Policy in Lucene controls how merging of segments is done.
The default since Solr/Lucene 3.3 is TieredMergePolicy.
The default since Lucene 2.3 was the LogByteSizeMergePolicy,
Even older versions of Lucene used LogDocMergePolicy.
-->
<!--
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<int name="maxMergeAtOnce">10</int>
<int name="segmentsPerTier">10</int>
</mergePolicy>
-->
<!-- Merge Factor
The merge factor controls how many segments will get merged at a time.
For TieredMergePolicy, mergeFactor is a convenience parameter which
will set both MaxMergeAtOnce and SegmentsPerTier at once.
For LogByteSizeMergePolicy, mergeFactor decides how many new segments
will be allowed before they are merged into one.
Default is 10 for both merge policies.
-->
<!--
<mergeFactor>10</mergeFactor>
-->
<!-- Expert: Merge Scheduler
The Merge Scheduler in Lucene controls how merges are
performed. The ConcurrentMergeScheduler (Lucene 2.3 default)
can perform merges in the background using separate threads.
The SerialMergeScheduler (Lucene 2.2 default) does not.
-->
<!--
<mergeScheduler
class="org.apache.lucene.index.ConcurrentMergeScheduler"/>
-->
----- Original Message -----
| From: "Radim Kolar" <h...@filez.com>
| To: solr-user@lucene.apache.org
| Sent: Saturday, October 27, 2012 7:44:46 PM
| Subject: Re: throttle segment merging
|
| Dne 26.10.2012 3:47, Tomás Fernández Löbbe napsal(a):
| >> Is there way to set-up logging to output something when segment
| >> merging
| >> runs?
| >>
| > I think segment merging is logged when you enable infoStream
| > logging (you
| > should see it commented in the solrconfig.xml)
| no, segment merging is not logged at info level. it needs customized
| log
| config.
|
| >
| >> Can be segment merges throttled?
| > You can change when and how segments are merged with the merge
| policy, maybe it's enough for you changing the initial settings
| (mergeFactor for example)?
|
| I am now researching elasticsearch, it can do it, its lucene 3.6
| based
|
