First of all, what I was seeing was different from what I thought I was seeing because a few weeks ago I uncommented the <autoCommit> block in the solrconfig.xml file and I didn't realize it until yesterday just before I went home, so that was controlling the commits more than the add and commit calls that I was making. When I commented that block out again, the times for index with add(docs, commitWithinMs) and with add(docs) and commit(false, false) were very similar. Both of them were about 20 minutes faster (38 minutes instead of about an hour) than indexing with <autoCommit> set to commit after every 1,000 documents or fifteen minutes.
Is this the blog post you are talking about: http://www.searchworkings.org/blog/-/blogs/gimme-all-resources-you-have-i-can-use-them!/? It seems to be about the right topic. I am using Solr 3.5. The feature matrix on one of the Lucid Imagination web pages says that DocumentWriterPerThread is available in Solr 4.0 and LucidWorks 2.0. I assume that means LucidWorks Enterprise. Is that right? Thanks, Mike -----Original Message----- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Thursday, April 05, 2012 2:45 PM To: solr-user@lucene.apache.org Subject: Re: waitFlush and waitSearcher with SolrServer.add(docs, commitWithinMs) Solr version? I suspect your outlier is due to merging segments, if so this should have happened quite some time into the run. See Simon Wilnauer's blog post on DocumenWriterPerThread (trunk) code. What commitWithin time are you using? Best Erick On Wed, Apr 4, 2012 at 7:50 PM, Mike O'Leary <tmole...@uw.edu> wrote: > I am indexing some database contents using add(docs, commitWithinMs), and > those add calls are taking over 80% of the time once the database begins > returning results. I was wondering if setting waitSearcher to false would > speed this up. Many of the calls take 1 to 6 seconds, with one outlier that > took over 11 minutes. > Thanks, > Mike > > -----Original Message----- > From: Mark Miller [mailto:markrmil...@gmail.com] > Sent: Wednesday, April 04, 2012 4:15 PM > To: solr-user@lucene.apache.org > Subject: Re: waitFlush and waitSearcher with SolrServer.add(docs, > commitWithinMs) > > > On Apr 4, 2012, at 6:50 PM, Mike O'Leary wrote: > >> If you index a set of documents with SolrJ and use >> StreamingUpdateSolrServer.add(Collection<SolrInputDocument> docs, int >> commitWithinMs), it will perform a commit within the time specified, and it >> seems to use default values for waitFlush and waitSearcher. >> >> Is there a place where you can specify different values for waitFlush >> and waitSearcher, or if you want to use different values do you have >> to call StreamingUpdateSolrServer.add(Collection<SolrInputDocument> >> docs) and then call StreamingUpdateSolrServer.commit(waitFlush, >> waitSearcher) explicitly? >> Thanks, >> Mike > > > waitFlush actually does nothing in recent versions of Solr. waitSearcher > doesn't seem so important when the commit is not done explicitly by the user > or a client. > > - Mark Miller > lucidimagination.com > > > > > > > > > > >
