Looking at it, and now knowing how much memory your other processes on your box use (nor how much memory you have set aside for Java), I would start with DOUBLING your ram. Make sure that you have enough Java memory.
You will know if it has some effect by using the 2:1 size ratio. 100mb for all that data ia pretty small, I think. Use the scientific method; Change only one parameter at a time and check results. It's always on of four things: (in different order depending on task, but listed alphabetically here) ------------------------------ Memory (process assigned and/or actual physical memory) Processor Network Bandwidth Hard Drive Bandwidth (sometimes you can add motherboard I/O paths also. as of this date, AMD has much more I/O paths in their consumer line of processors.) In order ease of experimenting with(Easiest to hardest): ----------------------------------- Appication/process assigned memory Physical memory Network Bandwidth HardDrive Bandwidth Screaming fast SCSI 15K rpm drives RAID arrays, casual RAID arrays, professional External DRAM drive 64 gig max/RAID them for more Processor(s) Put maximum speed/cache size motherboard will take. Otherwise, USUALLY requires changing motherboard/HOSTING setup I/O channels USUALLY requires changing motherboard/HOSTING setup Dennis Gearon Signature Warning ---------------- It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' EARTH has a Right To Life, otherwise we all die. --- On Sat, 10/9/10, sivaprasad <sivaprasa...@echidnainc.com> wrote: > From: sivaprasad <sivaprasa...@echidnainc.com> > Subject: Re: Speeding up solr indexing > To: solr-user@lucene.apache.org > Date: Saturday, October 9, 2010, 8:09 AM > > Hi, > Please find the configurations below. > > Machine configurations(Solr running here): > > RAM - 4 GB > HardDisk - 180GB > Os - Red Hat linux version 5 > Processor-2x Intel Core 2 Duo CPU @2.66GHz > > > > Machine configurations(Mysql server is running here): > RAM - 4 GB > HardDisk - 180GB > Os - Red Hat linux version 5 > Processor-2x Intel Core 2 Duo CPU @2.66GHz > > My sql Server deatils: > My sql version - Mysql 5.0.22 > > Solr configuration details: > > <indexDefaults> > > > <useCompoundFile>false</useCompoundFile> > > <mergeFactor>20</mergeFactor> > > > <!--<maxBufferedDocs>1000</maxBufferedDocs>--> > > > <ramBufferSizeMB>100</ramBufferSizeMB> > > <maxMergeDocs>2147483647</maxMergeDocs> > > <maxFieldLength>10000</maxFieldLength> > > <writeLockTimeout>1000</writeLockTimeout> > > <commitLockTimeout>10000</commitLockTimeout> > > <!--<luceneAutoCommit>false</luceneAutoCommit>--> > > > <!--<mergePolicy>org.apache.lucene.index.LogByteSizeMergePolicy</mergePolicy>--> > > > <!--<mergeScheduler>org.apache.lucene.index.ConcurrentMergeScheduler</mergeScheduler>--> > <lockType>single</lockType> > </indexDefaults> > > <mainIndex> > > > <useCompoundFile>false</useCompoundFile> > > <ramBufferSizeMB>100</ramBufferSizeMB> > <mergeFactor>20</mergeFactor> > > > <!--<maxBufferedDocs>1000</maxBufferedDocs>--> > > <maxMergeDocs>2147483647</maxMergeDocs> > > <maxFieldLength>10000</maxFieldLength> > > <unlockOnStartup>false</unlockOnStartup> > </mainIndex> > > <!-- the default high-performance update handler > --> > <updateHandler > class="solr.DirectUpdateHandler2"> > > <maxPendingDeletes>100000</maxPendingDeletes> > <autoCommit> > <maxDocs>10000</maxDocs> > <maxTime>60000</maxTime> > </autoCommit> > > <!-- A postCommit event is fired after > every commit or optimize command > <listener event="postCommit" > class="solr.RunExecutableListener"> > <str > name="exe">solr/bin/snapshooter</str> > <str name="dir">.</str> > <bool > name="wait">true</bool> > <arr name="args"> > <str>arg1</str> <str>arg2</str> > </arr> > <arr name="env"> > <str>MYVAR=val1</str> </arr> > </listener> > --> > <!-- A postOptimize event is fired only > after every optimize command, > useful > in conjunction with > index distribution to only distribute optimized > indicies > <listener event="postOptimize" > class="solr.RunExecutableListener"> > <str > name="exe">snapshooter</str> > <str > name="dir">solr/bin</str> > <bool > name="wait">true</bool> > </listener> > --> > </updateHandler> > > Solr document details: > > 21 fields are indexed and stored > 3 fileds are indexed only. > 3 fileds are stored only. > 3 fileds are indexed,stored and multi valued > 2 fileds indexed and multi valued > > And i am copying some of the indexed fileds.In this 2 > fileds are multivalued > and has thousands of values. > > In db-config-file the main table contains 0.6 million > records. > > When i tested for the same records, the index has taken 1hr > 30 min.In this > case one of the multivalued filed table doesn't have > records.After putting > data into this table,for each main table record , this > table has thousands > of records and this filed is indexed and stored.It is > taking more than 24 > hrs . > > Solr is running on tomcat 6.0.26, jdk1.6.0_17 and solr > 1.4.1 > > I am using JVM's default settings. > > Why this is taking this much time?Any body has suggestions, > where i am going > wrong. > > Thanks, > JS > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Speeding-up-solr-indexing-tp1667054p1670737.html > Sent from the Solr - User mailing list archive at > Nabble.com. >
