My first reaction is you have too much stuff on a single machine. Your cumulative index size is 2.4 TB. Granted, it's a beefy machine, but still...
And index size isn't all the helpful, as it includes the raw stored data which doesn't really come into play for sizing things, subtract out the *.fdt and *.fdx files. I'd use the expungeDelete and test. Actually, I'd move some of these cores to different machines. Best Erick On Fri, Oct 5, 2012 at 4:27 PM, jame vaalet <jamevaa...@gmail.com> wrote: > Hi Eric, > I am in a major dilemma with my index now. I have got 8 cores each around > 300 GB in size and half of them are deleted documents in it and above that > each has got around 100 segments as well. Do i issue a expungeDelete and > allow the merge policy to take care of the segments or optimize them into > single segment. Search performance is not at par compared to usual solr > speed. > If i have to optimize what segment number should i choose? my RAM size > around 120 GB and JVM heap is around 45 GB (oldGen being 30 GB). Pleas > advice ! > > thanks. > > > On 6 October 2012 00:00, Erick Erickson <erickerick...@gmail.com> wrote: > >> because eventually you'd run out of file handles. Imagine a >> long-running server with 100,000 segments. Totally >> unmanageable. >> >> I think shawn was emphasizing that RAM requirements don't >> depend on the number of segments. There are other >> resources that file consume however. >> >> Best >> Erick >> >> On Fri, Oct 5, 2012 at 1:08 PM, jame vaalet <jamevaa...@gmail.com> wrote: >> > hi Shawn, >> > thanks for the detailed explanation. >> > I have got one doubt, you said it doesn matter how many segments index >> have >> > but then why does solr has this merge policy which merges segments >> > frequently? why can it leave the segments as it is rather than merging >> > smaller one's into bigger one? >> > >> > thanks >> > . >> > >> > On 5 October 2012 05:46, Shawn Heisey <s...@elyograg.org> wrote: >> > >> >> On 10/4/2012 3:22 PM, jame vaalet wrote: >> >> >> >>> so imagine i have merged the 150 Gb index into single segment, this >> would >> >>> make a single segment of 150 GB in memory. When new docs are indexed it >> >>> wouldn't alter this 150 Gb index unless i update or delete the older >> docs, >> >>> right? will 150 Gb single segment have problem with memory swapping at >> OS >> >>> level? >> >>> >> >> >> >> Supplement to my previous reply: the real memory mentioned in the last >> >> paragraph does not include the memory that the OS uses to cache disk >> >> access. If more memory is needed and all the free memory is being used >> by >> >> the disk cache, the OS will throw away part of the disk cache (a >> >> near-instantaneous operation that should never involve disk I/O) and >> give >> >> that memory to the application that requests it. >> >> >> >> Here's a very good breakdown of how memory gets used with MMapDirectory >> in >> >> Solr. It's applicable to any program that uses memory mapping, not just >> >> Solr: >> >> >> >> http://java.dzone.com/**articles/use-lucene%E2%80%99s-**mmapdirectory< >> http://java.dzone.com/articles/use-lucene%E2%80%99s-mmapdirectory> >> >> >> >> Thanks, >> >> Shawn >> >> >> >> >> > >> > >> > -- >> > >> > -JAME >> > > > > -- > > -JAME
