Lance, makes sense and I have heard about the long GC times on large heaps but I personally haven't experienced a slowdown but that doesn't mean anything either :-). Agreed that tuning the SOLR caching is the way to go.
I haven't followed all the solr/lucene changes but from what I remember there are synchronization points that could be a bottleneck where adding more cores won't help this problem? Or am I completely missing something. Thanks again Amit On Mon, Aug 30, 2010 at 8:28 PM, scott chu (朱炎詹) <scott....@udngroup.com>wrote: > I am also curious as Amit does. Can you make an example about the garbage > collection problem you mentioned? > > ----- Original Message ----- From: "Lance Norskog" <goks...@gmail.com> > To: <solr-user@lucene.apache.org> > Sent: Tuesday, August 31, 2010 9:14 AM > Subject: Re: Hardware Specs Question > > > > It generally works best to tune the Solr caches and allocate enough >> RAM to run comfortably. Linux & Windows et. al. have their own cache >> of disk blocks. They use very good algorithms for managing this cache. >> Also, they do not make long garbage collection passes. >> >> On Mon, Aug 30, 2010 at 5:48 PM, Amit Nithian <anith...@gmail.com> wrote: >> >>> Lance, >>> >>> Thanks for your help. What do you mean by that the OS can keep the index >>> in >>> memory better than Solr? Do you mean that you should use another means to >>> keep the index in memory (i.e. ramdisk)? Is there a generally accepted >>> heap >>> size/index size that you follow? >>> >>> Thanks >>> Amit >>> >>> On Mon, Aug 30, 2010 at 5:00 PM, Lance Norskog <goks...@gmail.com> >>> wrote: >>> >>> The price-performance knee for small servers is 32G ram, 2-6 SATA >>>> disks on a raid, 8/16 cores. You can buy these servers and half-fill >>>> them, leaving room for expansion. >>>> >>>> I have not done benchmarks about the max # of processors that can be >>>> kept busy during indexing or querying, and the total numbers: QPS, >>>> response time averages & variability, etc. >>>> >>>> If your index file size is 8G, and your Java heap is 8G, you will do >>>> long garbage collection cycles. The operating system is very good at >>>> keeping your index in memory- better than Solr can. >>>> >>>> Lance >>>> >>>> On Mon, Aug 30, 2010 at 4:52 PM, Amit Nithian <anith...@gmail.com> >>>> wrote: >>>> > Hi all, >>>> > >>>> > I am curious to know get some opinions on at what point having more > >>>> CPU >>>> > cores shows diminishing returns in terms of QPS. Our index size is > >>>> about >>>> 8GB >>>> > and we have 16GB of RAM on a quad core 4 x 2.4 GHz AMD Opteron 2216. >>>> > Currently I have the heap to 8GB. >>>> > >>>> > We are looking to get more servers to increase capacity and because > >>>> the >>>> > warranty is set to expire on our old servers and so I was curious > >>>> before >>>> > asking for a certain spec what others run and at what point does > >>>> having >>>> more >>>> > cores cease to matter? Mainly looking at somewhere between 4-12 cores >>>> > per >>>> > server. >>>> > >>>> > Thanks! >>>> > Amit >>>> > >>>> >>>> >>>> >>>> -- >>>> Lance Norskog >>>> goks...@gmail.com >>>> >>>> >>> >> >> >> -- >> Lance Norskog >> goks...@gmail.com >> >> > > > -------------------------------------------------------------------------------- > > > > ___b___J_T_________f_r_C > Checked by AVG - www.avg.com > Version: 9.0.851 / Virus Database: 271.1.1/3102 - Release Date: 08/30/10 > 14:35:00 > >
