Erick, I actually agree and we are looking into bundling commits into a batch type update with soft-commits serving the batches and hard commit kicking in larger periods of time.
In practice, we have already noticed the periodic slow downs in search for exactly same queries before and after commit points. To describe it briefly: the queries that used to take lots of time to execute on solr 3.4 now execute super-fast whereas during the periodic slow downs they execute as slow as on solr 3.4. I bet there is a dependency, as you said, between the several searchers warmup and caches flushing. Thanks, Dmitry On Fri, Sep 27, 2013 at 3:44 AM, Erick Erickson <erickerick...@gmail.com>wrote: > Upping the number of concurrent warming searchers is almost always the > wrong thing to do. I'd lengthen the polling interval or the commit > interval. > Throwing away warming searchers is uselessly consuming resources. And > if you're trying to do any filter queries, your caches will almost never be > used since you're throwing them away so often. > > Best, > Erick > > On Thu, Sep 26, 2013 at 3:52 PM, Shawn Heisey <s...@elyograg.org> wrote: > > On 9/26/2013 10:56 AM, Dmitry Kan wrote: > >> > >> Btw, related to master-slave setup. What makes read-only slave not to > come > >> across the same issue? Would it not pull data from the master and warm > up > >> searchers? Or does it do updates in a more controlled fashion that makes > >> it > >> avoid these issues? > > > > > > Most people have the slave pollInterval configured on an interval that's > > pretty long, like 15 seconds to several minutes -- much longer than a > > typical searcher warming time. > > > > For a slave, new searchers are only created when there is a change copied > > over from the master. There may be several master-side commits that > happen > > during the pollInterval, but the slave won't see all of those. > > > > Thanks, > > Shawn > > >
