I actually think the example NRT setting of one second is probably lower than it should be.
When you think about most NRT cases, do you really need 1 second visibility? You normally could easily handle 10, 20, 30 seconds or more. Rather than just going for the low time, I think people should consider the highest they can go. You will get better performance, better caching, and in most use cases, no worse functionality. One second and less, IMO, should only be used when you critically require it for some esoteric reason. At least if you are designing for scale. For simple or small use cases, you might not worry about it. - Mark On Jan 10, 2013, at 2:33 PM, Upayavira <u...@odoko.co.uk> wrote: > Heh, the "it depends" answer :-) > > Thanks for the clarification. > > Upayavira > > On Thu, Jan 10, 2013, at 05:01 PM, Mark Miller wrote: >> I think it really depends - if you are gong for very fast visibility, >> your going to spend a bunch of time warming, and then just throw it out >> before it even gets much if any reuse. For very fast visibility >> turnaround, I suspect you don't want to do any warming. I think it >> depends on many things. >> >> One of the tradeoffs off using a very fast soft commit is that Sol's std >> caches will not be nearly as useful. >> >> - Mark >> >> On Jan 10, 2013, at 11:24 AM, Upayavira <u...@odoko.co.uk> wrote: >> >>> That's great Mark. Thx. One final question... all the stuff to do with >>> autowarming and static warming of caches - I presume all of that >>> configuration is still relevant (if less so) as you still need to warm >>> caches on a soft commit, even if those caches are much smaller than they >>> would be otherwise? >>> >>> Thanks! Upayavira >>> >>> On Thu, Jan 10, 2013, at 04:18 PM, Mark Miller wrote: >>>> There is no need to open a Searcher because you are controlling >>>> visibility through the faster 'soft' commit. That will reopen the reader >>>> from the IndexWriter. Because of that, there is no reason to do a heavy, >>>> non NRT Searcher reopen on hard commits. Essentially, the hard commit >>>> becomes simply about periodically flushing the tlog and the soft commit >>>> completely controls visibility. >>>> >>>> - Mark >>>> >>>> On Jan 10, 2013, at 9:41 AM, Upayavira <u...@odoko.co.uk> wrote: >>>> >>>>> And you don't need to open a searcher (openSearcher=false) because >>>>> you've got caches built up already alongside the in-memory NRT segment >>>>> which you can continue to use once the hard commit has happened? Is that >>>>> correct? >>>>> >>>>> (sorry for hijacking the thread - hopefully it is somewhat relevant) >>>>> >>>>> Upayavira >>>>> >>>>> On Thu, Jan 10, 2013, at 02:18 PM, Mark Miller wrote: >>>>>> Setup hard auto commit with openSeacher=false. I would do it at least >>>>>> once a minute. Don't worry about the commit being out of sync on the >>>>>> different nodes - you will be using soft commits for visibility. The hard >>>>>> commits will just be about relieving the pressure on the tlog. >>>>>> >>>>>> - Mark >>>>>> >>>>>> On Jan 10, 2013, at 6:43 AM, gadde <gadde....@gmail.com> wrote: >>>>>> >>>>>>> we have a SolrCloud with 3 nodes. we add documents to leader node and >>>>>>> use >>>>>>> commitwithin(100secs) option in SolrJ to add documents. AutoSoftCommit >>>>>>> in >>>>>>> SolrConfig is 1000ms. >>>>>>> >>>>>>> Transaction logs on replicas grew bigger than the index and we ran out >>>>>>> of >>>>>>> disk space in few days. Leader's tlogs are very small in few hundred >>>>>>> MBs. >>>>>>> >>>>>>> The following post suggest hard commit is required for "relieving the >>>>>>> memory >>>>>>> pressure of the transactionlog" >>>>>>> http://lucene.472066.n3.nabble.com/SolrCloud-is-softcommit-cluster-wide-for-the-collection-td4021584.html#a4021631 >>>>>>> >>>>>>> what is the best way to do a hard commit on this setup in SolrCloud? >>>>>>> >>>>>>> a. Through autoCommit in SolrConfig? which would cause hard commit on >>>>>>> all >>>>>>> the nodes at different times b. Trigger hard commit on leader while >>>>>>> updating >>>>>>> through SolrJ? >>>>>>> >>>>>>> Thanks >>>>>>> Shyam >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://lucene.472066.n3.nabble.com/SolrCloud-large-transaction-logs-tp4032160.html >>>>>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>>>> >>>> >>
