1- each document around 50 kb - 150 kb (web document) 2-final index is 40 gig 3-jre memory carefully given.
On Mon, Sep 26, 2011 at 9:57 PM, Jaeger, Jay - DOT <jay.jae...@dot.wi.gov>wrote: > 500 / second would be 1,800,000 per hour (much more than 500K documents). > > 1) how big is each document? > 2) how big are your index files? > 3) as others have recently written, make sure you don't give your JRE so > much memory that your OS is starved for memory to use for file system cache. > > JRJ > > -----Original Message----- > From: Lord Khan Han [mailto:khanuniver...@gmail.com] > Sent: Monday, September 26, 2011 6:09 AM > To: solr-user@lucene.apache.org > Subject: SOLR Index Speed > > Hi, > > We have 500K web document and usind solr (trunk) to index it. We have > special anaylizer which little bit heavy cpu . > Our machine config: > > 32 x cpu > 32 gig ram > SAS HD > > We are sending document with 16 reduce client (from hadoop) to the stand > alone solr server. the problem is we couldnt get speedier than the 500 doc > / > per sec. 500K document tooks 7-8 hours to index :( > > While indexin the the solr server cpu load is around : 5-6 (32 max) it > means %20 of the cpu total power. We have plenty ram ... > > I turned of auto commit and give 8198 rambuffer .. there is no io wait .. > > How can I make it faster ? > > PS: solr streamindex is not option because we need to submit javabin... > > thanks.. >
