Usability consideration, Not really answering your question, but i must comment using searching on items up to 100k makes faceted navigation very effective..but becomes least effective past 100k..u may want to consider breaking up the 500k documents in categories(typical breadcrumb) to 100k to faceted browse. Jeryl Cook
> To: solr-user@lucene.apache.org> From: [EMAIL PROTECTED]> Subject: Any tips > for indexing large amounts of data?> Date: Wed, 31 Oct 2007 10:30:50 -0400> > > Hi,> > I am creating an index of approx 500K documents. I wrote an indexing > > program using embeded solr: http://wiki.apache.org/solr/EmbeddedSolr > and am > seeing probably a 10 fold increase in indexing speeds. My > problem is > though, that if I try to reindex say 20K docs at a time it > slows down > considerably. I currently batch my updates in lots of 100 > and between > batches I close and reopen the "connection" to solr like so:> > private void > openConnection(String environment) throws > ParserConfigurationException, > IOException, SAXException {> System.setProperty("solr.solr.home", > SOLR_HOME);> solrConfig = new SolrConfig("solrconfig.xml");> solrCore = new > SolrCore(SOLR_HOME + "data/" + environment, > solrConfig, new > IndexSchema(solrConfig, "schema.xml"));> logger.debug("Opened solr > connection");> }> > private void closeConnection() {> solrCore.close();> > solrCore = null;> logger.debug("Closed solr connection");> }> > Does anyone > have any pointers or see anything obvious I'm doing wrong?> > Thanks> > > PS > Sorry if this is posted twice.
