TieredMergePolicy

2014-07-05 Thread Kireet Reddy
I have a question about the maxMergeAtOnce parameter. We are using elasticsearch and one of our nodes seems to have very high merge activity, However it seems to be high CPU activity and not I/O constrainted. I have enabled the IndexWriter info stream logs, and often times it seems to do merges

RE: TieredMergePolicy reclaimDeletesWeight

2013-06-19 Thread Petersen, Robert
Oh! Thanks for the info. I'll change that right away. -Original Message- From: Michael McCandless [mailto:luc...@mikemccandless.com] Sent: Wednesday, June 19, 2013 10:42 AM To: solr-user@lucene.apache.org Subject: Re: TieredMergePolicy reclaimDeletesWeight Way too high would cau

Re: TieredMergePolicy reclaimDeletesWeight

2013-06-19 Thread Michael McCandless
t: Wednesday, June 19, 2013 9:33 AM > To: solr-user@lucene.apache.org > Subject: Re: TieredMergePolicy reclaimDeletesWeight > > The default is 2.0, and higher values will more strongly favor merging > segments with deletes. > > I think 20.0 is likely way too high ... maybe

RE: TieredMergePolicy reclaimDeletesWeight

2013-06-19 Thread Petersen, Robert
Subject: Re: TieredMergePolicy reclaimDeletesWeight The default is 2.0, and higher values will more strongly favor merging segments with deletes. I think 20.0 is likely way too high ... maybe try 3-5? Mike McCandless http://blog.mikemccandless.com On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert

Re: TieredMergePolicy reclaimDeletesWeight

2013-06-19 Thread Michael McCandless
The default is 2.0, and higher values will more strongly favor merging segments with deletes. I think 20.0 is likely way too high ... maybe try 3-5? Mike McCandless http://blog.mikemccandless.com On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert wrote: > Hi > > In continuing a previous conve

TieredMergePolicy reclaimDeletesWeight

2013-06-18 Thread Petersen, Robert
Hi In continuing a previous conversation, I am attempting to not have to do optimizes on our continuously updated index in solr3.6.1 and I came across the mention of the reclaimDeletesWeight setting in this blog: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html We

Re: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-22 Thread Shawn Heisey
On 9/21/2011 4:22 PM, Michael Ryan wrote: I think the problem is that the config needs to be inside of the config, rather than after it as your have. Thank you, that took care of it. With mergeFactor set to 8 and the TMP options set to 35, it merges after 35 segments. Shawn

Re: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-22 Thread Michael McCandless
On Wed, Sep 21, 2011 at 10:10 PM, Michael Sokolov wrote: > I wonder if config-file validation would be helpful here :) I posted a patch > in SOLR-1758 once. Big +1. We should aim for as stringent config file checking as possible. Mike McCandless http://blog.mikemccandless.com

Re: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-21 Thread Michael Sokolov
I wonder if config-file validation would be helpful here :) I posted a patch in SOLR-1758 once. -Mike On 9/21/2011 6:22 PM, Michael Ryan wrote: I think the problem is that the config needs to be inside of the config, rather than after it as your have. -Michael

RE: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-21 Thread Michael Ryan
I think the problem is that the config needs to be inside of the config, rather than after it as your have. -Michael

Re: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-21 Thread Shawn Heisey
On 9/21/2011 11:18 AM, Shawn Heisey wrote: With no mergeFactor defined, maxMergeAtOnce and segmentsPerTier seem to be ignored. I've got both set to 35, but Solr is merging every 10 segments. I haven't tried explicitly setting mergeFactor yet to see if that will make the other settings overrid

Re: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-21 Thread Shawn Heisey
On 9/21/2011 3:10 PM, Chris Hostetter wrote: : With no mergeFactor defined, maxMergeAtOnce and segmentsPerTier seem to be : ignored. I've got both set to 35, but Solr is merging every 10 segments. I ... : Here's the relevant config pieces. These two sections are in separate files : inc

Re: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-21 Thread Chris Hostetter
: With no mergeFactor defined, maxMergeAtOnce and segmentsPerTier seem to be : ignored. I've got both set to 35, but Solr is merging every 10 segments. I ... : Here's the relevant config pieces. These two sections are in separate files : incorporated into solrconfig.xml using xinclude:

Re: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-21 Thread Shawn Heisey
On 9/20/2011 4:09 PM, Robert Muir wrote: yes, mergeFactory=10 is interpreted as both segmentsPerTier and maxMergeAtOnce. yes, specifying explicit TieredMP parameters will override whatever you set in mergeFactor (which is basically only interpreted to be backwards compatible) this is why i crea

Re: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-20 Thread Robert Muir
nd the > client lied to me about whether the message had been sent.) > > I'm still confused about the mergeFactor=10 setting in the example > configuration.  Took a quick look at the code, but I'm obviously looking in > the wrong place. Is mergeFactor=10 interpreted by T

RE: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-19 Thread Burton-West, Tom
l confused about the mergeFactor=10 setting in the example configuration. Took a quick look at the code, but I'm obviously looking in the wrong place. Is mergeFactor=10 interpreted by TieredMergePolicy as segmentsPerTier=10 and maxMergeAtOnce=10? If I specify values for these is the mergeFacto

Re: Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-16 Thread Robert Muir
On Fri, Sep 16, 2011 at 6:53 PM, Burton-West, Tom wrote: > Hello, > > The TieredMergePolicy has become the default with Solr 3.3, but the > configuration in the example uses the mergeFactor setting which applys to the > LogByteSizeMergePolicy. > > How is the mergeFact

Example setting TieredMergePolicy for Solr 3.3 or 3.4?

2011-09-16 Thread Burton-West, Tom
Hello, The TieredMergePolicy has become the default with Solr 3.3, but the configuration in the example uses the mergeFactor setting which applys to the LogByteSizeMergePolicy. How is the mergeFactor interpreted by the TieredMergePolicy? Is there an example somewhere showing how to configure

Example for Solr TieredMergePolicy configuration

2011-09-16 Thread Burton-West, Tom
Hello, The TieredMergePolicy has become the default with Solr 3.3, but the configuration in the example uses the mergeFactor setting which applys to the LogByteSizeMergePolicy. How is the mergeFactor interpreted by the TieredMergePolicy? Is there an example somewhere showing how to configure

Example configuring TieredMergePolicy in Solr

2011-09-16 Thread Burton-West, Tom
Hello, The TieredMergePolicy has become the default with Solr 3.3, but the configuration in the example uses the mergeFactor setting which applys to the LogByteSizeMergePolicy. How is the mergeFactor interpreted by the TieredMergePolicy? Is there an example somewhere showing how to configure