Re: Increasing maxMergedSegmentMB value

2016-01-31 Thread Zheng Lin Edwin Yeo
Hi Jack, Yes, I plan to merge all the 10GB segments to 20GB, and not just two of the segments. Sorry for the confusion. I have recently increased the system memory from 64GB to 192GB, but as our index size grows (which means more segments are created), I found that the query speed actually slow d

Re: Increasing maxMergedSegmentMB value

2016-01-31 Thread Jack Krupansky
Make sure you fully digest Mike McCandless' blog post on segment merge before trying to outguess his code: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html Generally, I don't think you would want to merge just two segments. Generally, you should do a bunch at a time,

Re: Increasing maxMergedSegmentMB value

2016-01-31 Thread Zheng Lin Edwin Yeo
Thanks for your reply Shawn and Jack. I wanted to increase the segment size to 15GB, so that there will be lesser segments to search for during the query, which should potentially improve the query speed. What if I set the segment size to 20GB? Will all the existing 10GB segments be merge to 20GB

Re: Increasing maxMergedSegmentMB value

2016-01-30 Thread Jack Krupansky
>From the Lucene MergePolicy Javadoc: "Whenever the segments in an index have been altered by IndexWriter , either the addition of a newly flushed segment, addition of many segments from addIndexes* calls, or a pre

Re: Increasing maxMergedSegmentMB value

2016-01-30 Thread Shawn Heisey
On 1/30/2016 7:31 AM, Zheng Lin Edwin Yeo wrote: > I would like to find out, when I increase the maxMergedSegmentMB from 10240 > (10GB) to 15360 (15GB), will all the 10GB segments that were created > previously be automatically merge to 15GB? Not necessarily. It will make those 10GB+ segments eli

Increasing maxMergedSegmentMB value

2016-01-30 Thread Zheng Lin Edwin Yeo
Hi, I would like to find out, when I increase the maxMergedSegmentMB from 10240 (10GB) to 15360 (15GB), will all the 10GB segments that were created previously be automatically merge to 15GB? This setting is part of my mergePolicy setting, which I am using TieredMergePolicy. 1