Ben, see also
http://www.nabble.com/Commit-in-solr-1.3-can-take-up-to-5-minutes-td19802781.html#a19802781
What type of physical drive is this and what interface is used (SATA, etc)?
What is the filesystem (NTFS)?
Did you add to an existing index from an older version of Solr, or
start from scratch?
If you add a single document to the index and commit, does it take a long time?
I notice your merge factor is 1000... this will create many files that
need to be sync'd
It may help to try the IndexWriter settings from the 1.3 example
setup... the important changes being:
<mergeFactor>10</mergeFactor>
<!--<maxBufferedDocs>1000</maxBufferedDocs>-->
<ramBufferSizeMB>32</ramBufferSizeMB>
-Yonik
On Mon, Sep 29, 2008 at 5:33 AM, Ben Shlomo, Yatir
<[EMAIL PROTECTED]> wrote:
> Hi!
>
>
>
> I am running on widows 64 bit ...
> I have upgraded to solr 1.3 in order to use the distributed search.
>
> I haven't changed the solrConfig and the schema xml files during the
> upgrade.
>
> I am indexing ~ 350K documents (each one is about 0.5 KB in size)
>
> The indexing takes a reasonable amount of time (350 seconds)
>
> See tomcat log:
>
> INFO: {add=[8x-wbTscWftuu1sVWpdnGw==, VOu1eSv0obBl1xkj2jGjIA==,
> YkOm-nKPrTVVVyeCZM4-4A==, rvaq_TyYsqt3aBc0KKDVbQ==,
> 9NdzWXsErbF_5btyT1JUjw==, ...(398728 more)]} 0 349875
>
>
>
> But when I commit it takes more than an hour ! (5000 seconds!, the
> optimize after the commit took 14 seconds)
>
> INFO: start commit(optimize=false,waitFlush=false,waitSearcher=true)
>
>
>
> p.s. its not a machine problem I moved to another machine and the same
> thing happened
>
>
> I noticed something very strange during the time I wait for the commit:
>
> While the solr index is 210MB in size
>
> In the windows task manager I noticed that the java process is making a
> HUGE amounts of IO reads:
>
> It reads more than 350 GB ! (- which takes a lot of time.)
>
> The process is constantly taking 25% of the cpu resources.
>
> All my autowarmCount in Solrconfig file do not exceed 256...
>
>
>
> Any more ideas to check?
>
> Thanks.
>
>
>
>
>
>
>
> Here is part of my solrConfig file:
>
> - <file:///C:\dss1\SolrHome\conf\solrconfig.xml##> < - <indexDefaults>
>
> - <!-- Values here affect all index writers and act as a default unless
> overridden.
>
> -->
>
> <useCompoundFile>false</useCompoundFile>
>
> <mergeFactor>1000</mergeFactor>
>
> <maxBufferedDocs>1000</maxBufferedDocs>
>
> <maxMergeDocs>2147483647</maxMergeDocs>
>
> <maxFieldLength>10000</maxFieldLength>
>
> <writeLockTimeout>1000</writeLockTimeout>
>
> <commitLockTimeout>10000</commitLockTimeout>
>
> </indexDefaults>
>
> - <mainIndex>
>
> - <!-- options specific to the main on-disk lucene index
>
> -->
>
> <useCompoundFile>false</useCompoundFile>
>
> <mergeFactor>1000</mergeFactor>
>
> <maxBufferedDocs>1000</maxBufferedDocs>
>
> <maxMergeDocs>2147483647</maxMergeDocs>
>
> <maxFieldLength>10000</maxFieldLength>
>
> - <!-- If true, unlock any held write or commit locks on startup.
>
> This defeats the locking mechanism that allows multiple
>
> processes to safely access a lucene index, and should be
>
> used with care.
>
> -->
>
> <unlockOnStartup>true</unlockOnStartup>
>
> </mainIndex>
>
>
>
>
>
>
>
>
>
>
>
> Yatir Ben-shlomo | eBay, Inc. | Classification Track, Shopping.com
> (Israel) | w: +972-9-892-1373 | email: [EMAIL PROTECTED] |
>
>
>
>
