500 / second would be 1,800,000 per hour (much more than 500K documents). 1) how big is each document? 2) how big are your index files? 3) as others have recently written, make sure you don't give your JRE so much memory that your OS is starved for memory to use for file system cache.
JRJ -----Original Message----- From: Lord Khan Han [mailto:khanuniver...@gmail.com] Sent: Monday, September 26, 2011 6:09 AM To: solr-user@lucene.apache.org Subject: SOLR Index Speed Hi, We have 500K web document and usind solr (trunk) to index it. We have special anaylizer which little bit heavy cpu . Our machine config: 32 x cpu 32 gig ram SAS HD We are sending document with 16 reduce client (from hadoop) to the stand alone solr server. the problem is we couldnt get speedier than the 500 doc / per sec. 500K document tooks 7-8 hours to index :( While indexin the the solr server cpu load is around : 5-6 (32 max) it means %20 of the cpu total power. We have plenty ram ... I turned of auto commit and give 8198 rambuffer .. there is no io wait .. How can I make it faster ? PS: solr streamindex is not option because we need to submit javabin... thanks..
