A major merge is an optimize. If you want to break up these major merges, there is an option to <commit> called 'maxSegments'. This lets you say "optimize a little". At least it makes stoppages controllable.
Merge time plotted against the number of documents is a fractal sawtooth. The degree of the fractal (kind of) comes from maxMergeDocs. It's a really hilarious graph at 1 billion documents. The function is the same as making change in currency of steady magnitude. From $1 to $9, it is steadily increasing, then drops to 1 at $10. And on upward logarthmically. Lance On Fri, Aug 20, 2010 at 12:57 PM, Devin Foley <devinfo...@gmail.com> wrote: > Thanks Yonik! I recognize your name from Googling for Solr issues. Your > help is greatly appreciated. > > I will attempt to tune maxMergeDocs as a first step. > > On Fri, Aug 20, 2010 at 12:39 PM, Yonik Seeley > <yo...@lucidimagination.com>wrote: > >> This sounds like perhaps a major merge was triggered. >> >> You could do a nightly optimize - which will take just as long, but >> you control when it happens. >> The other option is to prevent too big of segments being greated (at >> the expense of search speed) with options such as maxMergeDocs. >> >> -Yonik >> http://www.lucidimagination.com >> >> On Fri, Aug 20, 2010 at 2:03 PM, Devin Foley <devinfo...@gmail.com> wrote: >> > Hi All, >> > >> > I'm currently using Solr to run a search engine that is pulling in new >> data >> > 24 hours a day. Currently, I'm indexing about 5 million documents per >> day, >> > each document being around 2k in size. Every few days, the server locks >> up, >> > and calls to /update stop working. During this lock, CPU, memory and >> disk >> > utilization on the server shoot up. After 30-60 minutes, the server >> starts >> > accepting updates again, without intervention on my part. Also, during >> the >> > lock, I am still able to /select. I just can't /update. >> > >> > I viewed the thread dump in the admin panel during the last lockup, and >> > found a lot of these: >> > >> > 'btpool0-868' Id=7582, WAITING on >> > >> lock=java.util.concurrent.locks.reentrantreadwritelock$nonfairs...@6648711e >> , >> > total cpu time=0.0000ms user time=0.0000ms >> > ... (java.util stuff) >> > >> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:211) >> > >> > I'm using Solr 1.4.0, SolrJ 1.4.1 (StreamingUpdateSolrServer), and Java >> > 1.6.0 64bit on CentOS. >> > >> > My questions... >> > >> > 1) Has anybody else experienced this? >> > 2) Has this been addressed in Solr 1.4.0? >> > 3) How can I stop this from happening? >> > >> > Thanks! >> > Devin >> > >> > -- Lance Norskog goks...@gmail.com
