Hi Eric, I am in a major dilemma with my index now. I have got 8 cores each around 300 GB in size and half of them are deleted documents in it and above that each has got around 100 segments as well. Do i issue a expungeDelete and allow the merge policy to take care of the segments or optimize them into single segment. Search performance is not at par compared to usual solr speed. If i have to optimize what segment number should i choose? my RAM size around 120 GB and JVM heap is around 45 GB (oldGen being 30 GB). Pleas advice !
thanks. On 6 October 2012 00:00, Erick Erickson <erickerick...@gmail.com> wrote: > because eventually you'd run out of file handles. Imagine a > long-running server with 100,000 segments. Totally > unmanageable. > > I think shawn was emphasizing that RAM requirements don't > depend on the number of segments. There are other > resources that file consume however. > > Best > Erick > > On Fri, Oct 5, 2012 at 1:08 PM, jame vaalet <jamevaa...@gmail.com> wrote: > > hi Shawn, > > thanks for the detailed explanation. > > I have got one doubt, you said it doesn matter how many segments index > have > > but then why does solr has this merge policy which merges segments > > frequently? why can it leave the segments as it is rather than merging > > smaller one's into bigger one? > > > > thanks > > . > > > > On 5 October 2012 05:46, Shawn Heisey <s...@elyograg.org> wrote: > > > >> On 10/4/2012 3:22 PM, jame vaalet wrote: > >> > >>> so imagine i have merged the 150 Gb index into single segment, this > would > >>> make a single segment of 150 GB in memory. When new docs are indexed it > >>> wouldn't alter this 150 Gb index unless i update or delete the older > docs, > >>> right? will 150 Gb single segment have problem with memory swapping at > OS > >>> level? > >>> > >> > >> Supplement to my previous reply: the real memory mentioned in the last > >> paragraph does not include the memory that the OS uses to cache disk > >> access. If more memory is needed and all the free memory is being used > by > >> the disk cache, the OS will throw away part of the disk cache (a > >> near-instantaneous operation that should never involve disk I/O) and > give > >> that memory to the application that requests it. > >> > >> Here's a very good breakdown of how memory gets used with MMapDirectory > in > >> Solr. It's applicable to any program that uses memory mapping, not just > >> Solr: > >> > >> http://java.dzone.com/**articles/use-lucene%E2%80%99s-**mmapdirectory< > http://java.dzone.com/articles/use-lucene%E2%80%99s-mmapdirectory> > >> > >> Thanks, > >> Shawn > >> > >> > > > > > > -- > > > > -JAME > -- -JAME
