95+ % of the time problems like this are not Solr, but the data acquisition, i.e. querying the DB, traversing the file system etc.
We need to have an idea of what the indexing pipeline is all about before saying anything coherent. If you're using extractingrequesthandler for Word, PDFs, etc, you probably want to rack some clients together and move the parsing off to N clients. See: http://searchhub.org/2012/02/14/indexing-with-solrj/ This link also demonstrates using a JDBC connection too.. But I'm guessing here... Best, Erick On Tue, Jul 29, 2014 at 11:35 AM, Boogie Shafer <boogie.sha...@proquest.com> wrote: > when you say performance is very poor, what is happening at the system > level? > > e.g. > > are cpu's pegged out? > > is there a lot of IO wait? > > is the storage busy? > > is the network busy? > > > some easy tools to watch this stuff live if you arent sure and dont have > full on system monitoring agents installed > > nmon > htop > iftop > > the old standby sar/vmstat/iostat tools can get you the info in a less > "graphical" format > > ________________________________________ > From: Ameya Aware <ameya.aw...@gmail.com> > Sent: Tuesday, July 29, 2014 10:49 > To: solr-user@lucene.apache.org > Subject: Scaling Issues > > Hi, > > I am running Solr with below parameters: > > -XX:MaxPermSize=128m -Xms5120m -Xmx5120m -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=70 -XX:NewRatio=3 > -XX:MaxTenuringThreshold=8 -XX:+CMSParallelRemarkEnabled > -XX:+ParallelRefProcEnabled -XX:+UseLargePages -XX:+AggressiveOpts > -XX:-UseGCOverheadLimit > > > I need to index around 300000 documents but with above parameters > performance is coming very poor around 15000-20000 documents per hour. > > This would take a lot of time to index all the documents. > > Also, my autocommit in solrconfig.xml is as below: > > <autoCommit> > <maxTime>150000</maxTime> > <openSearcher>false</openSearcher> > </autoCommit> > > <documentCache class="solr.LRUCache" autowarmCount="0" initialSize="256" > size="256"/> > > > > I am running Solr on machine having 12GB RAM. > > > Please advice on how can i improve the performance. > > > Thanks, > Ameya >
