Thanks Erick for the explanation. I'll set my autocommit max time to 30 seconds then. But, I can let soft commit max time to 1/4 hour since it's an ads plateform which needs to be updated regularly.
2014-05-05 21:14 GMT+01:00 Erick Erickson <erickerick...@gmail.com>: > Take a look through the article I linked, 5 minutes may be an issue > since the transaction log will hold all 5 minutes worth of input. In > batch processes this can be quite a bit of data. Worse, when a Solr > instance terminates unexpectedly, the entire transaction log can be > replayed. > > Consider setting your autommit max time to something much shorter, say > 30 seconds. Or even less. NOTE openSearcher should be false. > > Then set your soft commit time to the latency you can stand, i.e. if > the users don't need to be able to search for a long time you can set > this to hours. > > FWIW, > Erick > > On Mon, May 5, 2014 at 11:03 AM, Hakim Benoudjit <h.benoud...@gmail.com> > wrote: > > I've tried it & it worked by letting solr do the commit instead of my > solr > > client. > > In solrconfig.xml: > > autocommit max_time has been set to 5 minutes & autosoftcommit max_time > to > > something bigger. > > > > Thanks a lot guys! > > > > > > 2014-05-05 16:30 GMT+01:00 Erick Erickson <erickerick...@gmail.com>: > > > >> You should not be committing from the client by and large, use the > >> <autoCommit> and <autoSoftCommit> options in solrconfig.xml. > >> > >> See: > >> > http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ > >> > >> Best, > >> Erick > >> > >> On Mon, May 5, 2014 at 8:12 AM, Hakim Benoudjit <h.benoud...@gmail.com> > >> wrote: > >> > Is there an option in Solr (solrconfig.xml or somewhere else) to > >> regularize > >> > commits to the index. > >> > I meant to do a 'sleep' between each commit to the index, when data > >> > to-be-indexed is waiting inside a stack. > >> > > >> > > >> > 2014-05-05 15:58 GMT+01:00 Hakim Benoudjit <h.benoud...@gmail.com>: > >> > > >> >> The index is made with the same version of solr, that is searching > >> >> (4.6.0), the config file (solrconfig.xml) & schema.xml is the same > too. > >> >> The only way for me to solve this issue is to let only one process to > >> >> index at the same time. Wouldnt a layer of message queue resolve this > >> issue? > >> >> > >> >> > >> >> 2014-05-04 18:33 GMT+01:00 Shawn Heisey <s...@elyograg.org>: > >> >> > >> >> On 5/4/2014 9:30 AM, Hakim Benoudjit wrote: > >> >>> > Ok. These files contain what you've requested: > >> >>> > > >> >>> > First (the xml error): http://pastebin.com/ZcagK3T7 > >> >>> > Second (java params): http://pastebin.com/JtWQpp6s > >> >>> > Third (Solr version): http://pastebin.com/wYdpdsAW > >> >>> > >> >>> Are you running with an index originally built by an earlier > version of > >> >>> Solr? If you are, you may be running into a known bug. The last > >> >>> "caused by" section of the java stacktrace looks similar to the one > in > >> >>> this issue -- which is indeed index corruption: > >> >>> > >> >>> https://issues.apache.org/jira/browse/LUCENE-5377 > >> >>> > >> >>> If that's the problem you're experiencing, upgrading your Solr > version > >> >>> will hopefully fix it. Simply dropping in the 4.6.1 war file and > any > >> >>> contrib jars should cause zero problems for your 4.6.0 install. > >> >>> Upgrading to 4.7.2 or 4.8.0 should be done with more care. > >> >>> > >> >>> Thanks, > >> >>> Shawn > >> >>> > >> >>> > >> >> > >> >> > >> >> -- > >> >> Hakim Benoudjit. > >> >> > >> > > >> > > >> > > >> > -- > >> > Hakim Benoudjit. > >> > > > > > > > > -- > > Hakim Benoudjit. > -- Hakim Benoudjit.
