Take a look through the article I linked, 5 minutes may be an issue since the transaction log will hold all 5 minutes worth of input. In batch processes this can be quite a bit of data. Worse, when a Solr instance terminates unexpectedly, the entire transaction log can be replayed.
Consider setting your autommit max time to something much shorter, say 30 seconds. Or even less. NOTE openSearcher should be false. Then set your soft commit time to the latency you can stand, i.e. if the users don't need to be able to search for a long time you can set this to hours. FWIW, Erick On Mon, May 5, 2014 at 11:03 AM, Hakim Benoudjit <h.benoud...@gmail.com> wrote: > I've tried it & it worked by letting solr do the commit instead of my solr > client. > In solrconfig.xml: > autocommit max_time has been set to 5 minutes & autosoftcommit max_time to > something bigger. > > Thanks a lot guys! > > > 2014-05-05 16:30 GMT+01:00 Erick Erickson <erickerick...@gmail.com>: > >> You should not be committing from the client by and large, use the >> <autoCommit> and <autoSoftCommit> options in solrconfig.xml. >> >> See: >> http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ >> >> Best, >> Erick >> >> On Mon, May 5, 2014 at 8:12 AM, Hakim Benoudjit <h.benoud...@gmail.com> >> wrote: >> > Is there an option in Solr (solrconfig.xml or somewhere else) to >> regularize >> > commits to the index. >> > I meant to do a 'sleep' between each commit to the index, when data >> > to-be-indexed is waiting inside a stack. >> > >> > >> > 2014-05-05 15:58 GMT+01:00 Hakim Benoudjit <h.benoud...@gmail.com>: >> > >> >> The index is made with the same version of solr, that is searching >> >> (4.6.0), the config file (solrconfig.xml) & schema.xml is the same too. >> >> The only way for me to solve this issue is to let only one process to >> >> index at the same time. Wouldnt a layer of message queue resolve this >> issue? >> >> >> >> >> >> 2014-05-04 18:33 GMT+01:00 Shawn Heisey <s...@elyograg.org>: >> >> >> >> On 5/4/2014 9:30 AM, Hakim Benoudjit wrote: >> >>> > Ok. These files contain what you've requested: >> >>> > >> >>> > First (the xml error): http://pastebin.com/ZcagK3T7 >> >>> > Second (java params): http://pastebin.com/JtWQpp6s >> >>> > Third (Solr version): http://pastebin.com/wYdpdsAW >> >>> >> >>> Are you running with an index originally built by an earlier version of >> >>> Solr? If you are, you may be running into a known bug. The last >> >>> "caused by" section of the java stacktrace looks similar to the one in >> >>> this issue -- which is indeed index corruption: >> >>> >> >>> https://issues.apache.org/jira/browse/LUCENE-5377 >> >>> >> >>> If that's the problem you're experiencing, upgrading your Solr version >> >>> will hopefully fix it. Simply dropping in the 4.6.1 war file and any >> >>> contrib jars should cause zero problems for your 4.6.0 install. >> >>> Upgrading to 4.7.2 or 4.8.0 should be done with more care. >> >>> >> >>> Thanks, >> >>> Shawn >> >>> >> >>> >> >> >> >> >> >> -- >> >> Hakim Benoudjit. >> >> >> > >> > >> > >> > -- >> > Hakim Benoudjit. >> > > > > -- > Hakim Benoudjit.
