Re: Any tips for indexing large amounts of data?

[scott.tabar]

Greetings Brendan,

In the solrconfig.xml file, under the updateHandler, is an auto commit 
statement.

It looks like:

    <autoCommit> 
      <maxDocs>1000</maxDocs>
      <maxTime>1000</maxTime>
    </autoCommit>

I would think you would see better performance by allowing auto commit to 
handle the commit size instead of reopening the connection all the time.

I believe the maxTime is in milliseconds.

Let us know if this helps,
   Scott Tabar

---- Brendan Grainger <[EMAIL PROTECTED]> wrote: 
Hi,

I am creating an index of approx 500K documents. I wrote an indexing  
program using embeded solr: http://wiki.apache.org/solr/EmbeddedSolr  
and am seeing probably a 10 fold increase in indexing speeds. My  
problem is though, that if I try to reindex say 20K docs at a time it  
slows down considerably. I currently batch my updates in lots of 100  
and between batches I close and reopen the "connection" to solr like so:

     private void openConnection(String environment) throws  
ParserConfigurationException, IOException, SAXException {
         System.setProperty("solr.solr.home", SOLR_HOME);
         solrConfig = new SolrConfig("solrconfig.xml");
         solrCore = new SolrCore(SOLR_HOME + "data/" + environment,  
solrConfig, new IndexSchema(solrConfig, "schema.xml"));
         logger.debug("Opened solr connection");
     }

     private void closeConnection() {
         solrCore.close();
         solrCore = null;
         logger.debug("Closed solr connection");
     }

Does anyone have any pointers or see anything obvious I'm doing wrong?

Thanks


PS Sorry if this is posted twice.