On Mon, Oct 19, 2009 at 7:39 PM, Lance Norskog <goks...@gmail.com> wrote: > commit(waitFlush="true", waitSearcher="true") waits for the entire > operation and when it finishes, all 1 million documents should be > searchable.
That waits for the commit to complete, but not any adds that may be happening in parallel (that's pretty much impossible). If the client uses multiple threads to do adds, it's currently the responsibility of the client to wait for all of the pending adds to complete before calling commit. -Yonik http://www.lucidimagination.com > Please try this same test with Solr 1.4 and post your results. To make > it easier, here is the first release candidate: > > http://people.apache.org/~gsingers/solr/1.4.0-RC/ > > > On Mon, Oct 19, 2009 at 1:06 PM, SharmilaR > <sranganat...@library.rochester.edu> wrote: >> >> Solr version is 1.3 >> I am indexing total of 1.4 million documents. Yes, I commit(waitFlush="true" >> waitSearcher="true") every 100k documents and then one at the end. >> I have a counter next to addDoc(SolrDocument) statement to keep track of >> number of documents added. When I query Solr after commit, the total number >> of documents returned does not match the number of documents added. This >> happens only when I index millions of documents and not when I index like >> 500 documents. In this case, I know its the last 20 documents which are not >> committed because each document has a field 'RECORD_ID' which is assigned >> sequential number(in java code). When I query Solr using Solr admin >> interface, the documents with last 20 RECORD_ID are missing.(example the >> last id is 999,980 instead of 1,000,000) >> >> - Sharmila >> >> >> Feak, Todd wrote: >>> >>> A few questions to help the troubleshooting. >>> >>> Solr version #? >>> >>> Is there just 1 commit through Solrj for the millions of documents? >>> >>> Or do you do it on a regular interval (every 100k documents for example) >>> and then one at the end to be sure? >>> >>> How are you observing that the last few didn't make it in? Are you looking >>> at a slave or master? >>> >>> -Todd >>> >>> >> -----Original Message----- >> From: Ranganathan, Sharmila [mailto:sranganat...@library.rochester.edu] >> Sent: Monday, October 19, 2009 9:19 AM >> To: solr-user@lucene.apache.org >> Subject: Solr commits before documents are added >> >> Hi, >> >> My application indexes huge number of documents(like in millions). Below >> is the snapshot of my code where I add all documents to Solr, and then >> at last issue commit command. I use Solrj. I find that last few >> documents are not committed to Solr. Is this because adding documents >> to Solr took longer time and it reached commit command even before it >> finished adding documents? Is there are way to ensure that solr waits >> for all documents to be added and then commits? Please advise me how to >> solve this issue. >> >> >> >> For loop >> >> solrServer.add(doc); // Add document to Solr >> >> End for loop >> >> solrServer.commit(); // Commit to Solr >> >> >> >> >> >> Thanks, >> >> Sharmila >> >> >> >> >> >> -- >> View this message in context: >> http://www.nabble.com/Solr-commits-before-documents-are-added-tp25961191p25964770.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > Lance Norskog > goks...@gmail.com >
