Just to open the can of worms, it *can* be possible to have very low commit times, we have 250ms currently and are in production with that. But it does come with pain (no such thing as a free lunch!), we had to turn off ALL the Solr caches (warming is useless at that kind of frequency, it will take longer to warm the cache than the time before the next commit), and throw a lot of RAM and expensive SSDs at the problem.
That said, Shawn's advice is correct, anything less than 1s commit shouldn't be needed for most users, and I would concur with staying away from it unless you absolutely decide you have to have it. You only go that route if you are prepared to commit (no pun intended!) a fair amount of time, money and resources to investigating and dealing with issues. We will have a talk at Revolution this year about some of the scale and latency issues we have to deal with (blatant plug for my team lead who's giving the talk!) On 17 August 2015 at 14:31, Shawn Heisey <apa...@elyograg.org> wrote: > On 8/17/2015 7:04 AM, Maulin Rathod wrote: > > We have observed that Intermittently querying become slower when > documentCache become empty. The documentCache is getting flushed whenever > new document added to the collection. > > > > Is there any way by which we can ensure that newly added documents are > visible without losing data in documentCache? We are trying to use soft > commit but it also flushes all documents in documentCache. > > <snip> > > > <autoSoftCommit> > > <maxTime>50</maxTime> > > </autoSoftCommit> > > You are doing a soft commit within 50 milliseconds of adding a new > document. Solr can have severe performance problems when autoSoftCommit > is set to 1000 -- one second. 50 milliseconds is one twentieth of a > very low value that is known to cause problems. It can make the problem > much more than 20 times worse. > > Please read this article: > > > http://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ > > Note one particular section, which says the following: Don’t listen to > your product manager who says "we need no more than 1 second latency". > > You need to set your commit interval as long as you possibly can. I > personally wouldn't go longer than 60 seconds, 30 seconds if the commits > complete particularly fast. It should be several minutes if that will > meet your needs. When your commit interval is very low, Solr's caches > can become useless, as you've noticed. > > TL;DR info: Your autoCommit settings have openSearcher set to false, so > they do not matter for the problem you have described. I would probably > increase that to 5 minutes rather than 15 seconds, but that is not very > important here, and 15 seconds for hard commits that don't open a new > searcher is known to have a low impact on performance. "Low" impact > isn't the same as NO impact, so I keep this interval long as well. > > Thanks, > Shawn > >
