Thanks, Svante. Your indexing speed using db seems to really fast. Can you please provide some more detail on how you are indexing db records. Is it thru DataImportHandler? And what database? Is that local db? We are indexing around 70 fields (60 multivalued) but data is not populated always in all fields. The average size of document is in 5-10 kbs.
-----Original Message----- From: saka.csi...@gmail.com [mailto:saka.csi...@gmail.com] On Behalf Of svante karlsson Sent: Friday, January 24, 2014 5:05 PM To: solr-user@lucene.apache.org Subject: Re: Solr server requirements for 100+ million documents I just indexed 100 million db docs (records) with 22 fields (4 multivalued) in 9524 sec using libcurl. 11 million took 763 seconds so the speed drops somewhat with increasing dbsize. We write 1000 docs (just an arbitrary number) in each request from two threads. If you will be using solrcloud you will want more writer threads. The hardware is a single cheap hp DL320E GEN8 V2 1P E3-1220V3 with one SSD and 32GB and the solr runs on ubuntu 13.10 inside a esxi virtual machine. /svante 2014/1/24 Susheel Kumar <susheel.ku...@thedigitalgroup.net> > Thanks, Erick for the info. > > For indexing I agree the more time is consumed in data acquisition > which in our case from Database. For indexing currently we are using > the manual process i.e. Solr dashboard Data Import but now looking to > automate. How do you suggest to automate the index part. Do you > recommend to use SolrJ or should we try to automate using Curl? > > > -----Original Message----- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Friday, January 24, 2014 2:59 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr server requirements for 100+ million documents > > Can't be done with the information you provided, and can only be > guessed at even with more comprehensive information. > > Here's why: > > > http://searchhub.org/2012/07/23/sizing-hardware-in-the-abstract-why-we > -dont-have-a-definitive-answer/ > > Also, at a guess, your indexing speed is so slow due to data > acquisition; I rather doubt you're being limited by raw Solr indexing. > If you're using SolrJ, try commenting out the > server.add() bit and running again. My guess is that your indexing > speed will be almost unchanged, in which case it's the data > acquisition process is where you should concentrate efforts. As a > comparison, I can index 11M Wikipedia docs on my laptop in 45 minutes > without any attempts at parallelization. > > > Best, > Erick > > On Fri, Jan 24, 2014 at 12:10 PM, Susheel Kumar < > susheel.ku...@thedigitalgroup.net> wrote: > > Hi, > > > > Currently we are indexing 10 million document from database (10 db > > data > entities) & index size is around 8 GB on windows virtual box. Indexing > in one shot taking 12+ hours while indexing parallel in separate cores > & merging them together taking 4+ hours. > > > > We are looking to scale to 100+ million documents and looking for > recommendation on servers requirements on below parameters for a > Production environment. There can be 200+ users performing search same time. > > > > No of physical servers (considering solr cloud) Memory requirement > > Processor requirement (# cores) Linux as OS oppose to windows > > > > Thanks in advance. > > Susheel > > >
