commit(waitFlush="true", waitSearcher="true") waits for the entire operation and when it finishes, all 1 million documents should be searchable.
Please try this same test with Solr 1.4 and post your results. To make it easier, here is the first release candidate: http://people.apache.org/~gsingers/solr/1.4.0-RC/ On Mon, Oct 19, 2009 at 1:06 PM, SharmilaR <sranganat...@library.rochester.edu> wrote: > > Solr version is 1.3 > I am indexing total of 1.4 million documents. Yes, I commit(waitFlush="true" > waitSearcher="true") every 100k documents and then one at the end. > I have a counter next to addDoc(SolrDocument) statement to keep track of > number of documents added. When I query Solr after commit, the total number > of documents returned does not match the number of documents added. This > happens only when I index millions of documents and not when I index like > 500 documents. In this case, I know its the last 20 documents which are not > committed because each document has a field 'RECORD_ID' which is assigned > sequential number(in java code). When I query Solr using Solr admin > interface, the documents with last 20 RECORD_ID are missing.(example the > last id is 999,980 instead of 1,000,000) > > - Sharmila > > > Feak, Todd wrote: >> >> A few questions to help the troubleshooting. >> >> Solr version #? >> >> Is there just 1 commit through Solrj for the millions of documents? >> >> Or do you do it on a regular interval (every 100k documents for example) >> and then one at the end to be sure? >> >> How are you observing that the last few didn't make it in? Are you looking >> at a slave or master? >> >> -Todd >> >> > -----Original Message----- > From: Ranganathan, Sharmila [mailto:sranganat...@library.rochester.edu] > Sent: Monday, October 19, 2009 9:19 AM > To: solr-user@lucene.apache.org > Subject: Solr commits before documents are added > > Hi, > > My application indexes huge number of documents(like in millions). Below > is the snapshot of my code where I add all documents to Solr, and then > at last issue commit command. I use Solrj. I find that last few > documents are not committed to Solr. Is this because adding documents > to Solr took longer time and it reached commit command even before it > finished adding documents? Is there are way to ensure that solr waits > for all documents to be added and then commits? Please advise me how to > solve this issue. > > > > For loop > > solrServer.add(doc); // Add document to Solr > > End for loop > > solrServer.commit(); // Commit to Solr > > > > > > Thanks, > > Sharmila > > > > > > -- > View this message in context: > http://www.nabble.com/Solr-commits-before-documents-are-added-tp25961191p25964770.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- Lance Norskog goks...@gmail.com
