Thx, that makes a lot of sense. Upayavira
On Thu, Jan 10, 2013, at 09:37 PM, Mark Miller wrote: > I actually think the example NRT setting of one second is probably lower > than it should be. > > When you think about most NRT cases, do you really need 1 second > visibility? You normally could easily handle 10, 20, 30 seconds or more. > Rather than just going for the low time, I think people should consider > the highest they can go. You will get better performance, better caching, > and in most use cases, no worse functionality. > > One second and less, IMO, should only be used when you critically require > it for some esoteric reason. > > At least if you are designing for scale. For simple or small use cases, > you might not worry about it. > > - Mark > > On Jan 10, 2013, at 2:33 PM, Upayavira <u...@odoko.co.uk> wrote: > > > Heh, the "it depends" answer :-) > > > > Thanks for the clarification. > > > > Upayavira > > > > On Thu, Jan 10, 2013, at 05:01 PM, Mark Miller wrote: > >> I think it really depends - if you are gong for very fast visibility, > >> your going to spend a bunch of time warming, and then just throw it out > >> before it even gets much if any reuse. For very fast visibility > >> turnaround, I suspect you don't want to do any warming. I think it > >> depends on many things. > >> > >> One of the tradeoffs off using a very fast soft commit is that Sol's std > >> caches will not be nearly as useful. > >> > >> - Mark > >> > >> On Jan 10, 2013, at 11:24 AM, Upayavira <u...@odoko.co.uk> wrote: > >> > >>> That's great Mark. Thx. One final question... all the stuff to do with > >>> autowarming and static warming of caches - I presume all of that > >>> configuration is still relevant (if less so) as you still need to warm > >>> caches on a soft commit, even if those caches are much smaller than they > >>> would be otherwise? > >>> > >>> Thanks! Upayavira > >>> > >>> On Thu, Jan 10, 2013, at 04:18 PM, Mark Miller wrote: > >>>> There is no need to open a Searcher because you are controlling > >>>> visibility through the faster 'soft' commit. That will reopen the reader > >>>> from the IndexWriter. Because of that, there is no reason to do a heavy, > >>>> non NRT Searcher reopen on hard commits. Essentially, the hard commit > >>>> becomes simply about periodically flushing the tlog and the soft commit > >>>> completely controls visibility. > >>>> > >>>> - Mark > >>>> > >>>> On Jan 10, 2013, at 9:41 AM, Upayavira <u...@odoko.co.uk> wrote: > >>>> > >>>>> And you don't need to open a searcher (openSearcher=false) because > >>>>> you've got caches built up already alongside the in-memory NRT segment > >>>>> which you can continue to use once the hard commit has happened? Is that > >>>>> correct? > >>>>> > >>>>> (sorry for hijacking the thread - hopefully it is somewhat relevant) > >>>>> > >>>>> Upayavira > >>>>> > >>>>> On Thu, Jan 10, 2013, at 02:18 PM, Mark Miller wrote: > >>>>>> Setup hard auto commit with openSeacher=false. I would do it at least > >>>>>> once a minute. Don't worry about the commit being out of sync on the > >>>>>> different nodes - you will be using soft commits for visibility. The > >>>>>> hard > >>>>>> commits will just be about relieving the pressure on the tlog. > >>>>>> > >>>>>> - Mark > >>>>>> > >>>>>> On Jan 10, 2013, at 6:43 AM, gadde <gadde....@gmail.com> wrote: > >>>>>> > >>>>>>> we have a SolrCloud with 3 nodes. we add documents to leader node and > >>>>>>> use > >>>>>>> commitwithin(100secs) option in SolrJ to add documents. > >>>>>>> AutoSoftCommit in > >>>>>>> SolrConfig is 1000ms. > >>>>>>> > >>>>>>> Transaction logs on replicas grew bigger than the index and we ran > >>>>>>> out of > >>>>>>> disk space in few days. Leader's tlogs are very small in few hundred > >>>>>>> MBs. > >>>>>>> > >>>>>>> The following post suggest hard commit is required for "relieving the > >>>>>>> memory > >>>>>>> pressure of the transactionlog" > >>>>>>> http://lucene.472066.n3.nabble.com/SolrCloud-is-softcommit-cluster-wide-for-the-collection-td4021584.html#a4021631 > >>>>>>> > >>>>>>> what is the best way to do a hard commit on this setup in SolrCloud? > >>>>>>> > >>>>>>> a. Through autoCommit in SolrConfig? which would cause hard commit on > >>>>>>> all > >>>>>>> the nodes at different times b. Trigger hard commit on leader while > >>>>>>> updating > >>>>>>> through SolrJ? > >>>>>>> > >>>>>>> Thanks > >>>>>>> Shyam > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> View this message in context: > >>>>>>> http://lucene.472066.n3.nabble.com/SolrCloud-large-transaction-logs-tp4032160.html > >>>>>>> Sent from the Solr - User mailing list archive at Nabble.com. > >>>>>> > >>>> > >> >
