I think this is expected. As Shawn mentioned, your hard commits have openSearcher=false, so they flush changes to disk, but don't force a re-open of the active searcher. By contrast softCommit, sets openSearcher=true, the point of softCommit is to make the changes visible so do to that you have to re-open a searcher.
Currently, most of the Solr Caches are searcher-based, so opening a new searcher means creating (and optionally warming) a new cache. I know there is work in progress to make these caches more segment-based (which would lead to more re-use between searchers) but currently each commit would create a new set of caches. You can "warm" those caches, but that's where the trade-off comes, warming a cache with records takes time and processing, so if you are committing frequently, the warming doesn't really have time to take effect. https://wiki.apache.org/solr/SolrCaching explains this (probably better than me) and from that page, the DocumentCache can't be auto-warmed since DocIds can change between searchers. On 18 August 2015 at 06:19, Maulin Rathod <mrat...@asite.com> wrote: > Hi Shawn, > > > > Thanks for your feedback. > > In our scenario documents are added frequently (Approx 10 documents added > in 1 minute) and we want to make it available for search near realtime > (within 5 second). Even if we set autosoftcommit 5 second (so that > document will be available for search after 5 second), it flushes all > documents from documentCache. Just wanted to understand if we are doing > something wrong or its solr expected behavior. > > > > > > <autoSoftCommit> > > <maxTime>5000</maxTime> > > </autoSoftCommit> > > > > > > Regards, > > > > Maulin > > > > > > > > -----Original Message----- > From: Shawn Heisey [mailto:apa...@elyograg.org] > Sent: 17 August 2015 19:02 > To: solr-user@lucene.apache.org > Subject: Re: Solr Caching (documentCache) not working > > > > On 8/17/2015 7:04 AM, Maulin Rathod wrote: > > > We have observed that Intermittently querying become slower when > documentCache become empty. The documentCache is getting flushed whenever > new document added to the collection. > > > > > > Is there any way by which we can ensure that newly added documents are > visible without losing data in documentCache? We are trying to use soft > commit but it also flushes all documents in documentCache. > > > > <snip> > > > > > <autoSoftCommit> > > > <maxTime>50</maxTime> > > > </autoSoftCommit> > > > > You are doing a soft commit within 50 milliseconds of adding a new > document. Solr can have severe performance problems when autoSoftCommit is > set to 1000 -- one second. 50 milliseconds is one twentieth of a very low > value that is known to cause problems. It can make the problem much more > than 20 times worse. > > > > Please read this article: > > > > > http://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ > > > > Note one particular section, which says the following: Don’t listen to > your product manager who says "we need no more than 1 second latency". > > > > You need to set your commit interval as long as you possibly can. I > personally wouldn't go longer than 60 seconds, 30 seconds if the commits > complete particularly fast. It should be several minutes if that will meet > your needs. When your commit interval is very low, Solr's caches can > become useless, as you've noticed. > > > > TL;DR info: Your autoCommit settings have openSearcher set to false, so > they do not matter for the problem you have described. I would probably > increase that to 5 minutes rather than 15 seconds, but that is not very > important here, and 15 seconds for hard commits that don't open a new > searcher is known to have a low impact on performance. "Low" impact isn't > the same as NO impact, so I keep this interval long as well. > > > > Thanks, > > Shawn > > >
