Oh dear. Your autowarming is almost, but not quite totally, useless given your 2 second soft commit interval. See: https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
So autowarming is probably not a cure, when you originally said “commit” I was assuming that was one that opened a new searcher, if that’s not true autowarming isn’t a cure. Do you _really_ require 2 second soft commit intervals? I would not be surprised if you also see “too many on deck searcher” warnings in your logs at times. This is one of my hot buttons, having very short soft commit intervals is something people do without understanding the tradeoffs, one of which is that your caches are probably getting a poor utilization rate. Often the recommendation for short intervals like this is to not use the caches at all. The newSearcher is a full query. Go ahead and add facets. But again, this probably isn’t going to help much. But really, revisit your autocommit settings. Taking 1.7 seconds to autowarm means that you have roughly this. - commit - 1.7 seconds later, the new searcher is open for business. - 0.3 seconds after that a new searcher is open, which takes another 1.7 seconds to autowarm. I doubt your hard commit is really the culprit here _unless_ you’re running on an under-powered machine. The hard commit will trigger segment merging, which is CPU and I/O intensive. If you’re using a machine that can’t afford the cycles to be taken up by merging, that could account for what you see, but new searchers are being opened every 2 seconds (assuming a relatively constant indexing load). Best, Erick > On Mar 30, 2020, at 6:42 PM, Revas <revas2...@gmail.com> wrote: > > Thanks, Erick, > > The process time execution based on debugQuery between the query and facets > is as follows > > query 10ms > facets 4900ms > > since max time is spent on facet processing (docValues enabled), query and > filter cache do no apply to this, correct? > > > - Autowarm count is at 32 for both and autowarm time is 25 for > query-result cache and 1724 for filter cache > - We have hard-commit every 5 mins with opensearcher=false and > soft-commit every 2 secs. > - facet are a mix of pivot facets,range facets and facet queries > - when the same facets criteria bring a smaller result set, response is > much faster > > > > > On Mon, Mar 30, 2020 at 4:47 PM Erick Erickson <erickerick...@gmail.com> > wrote: > >> OK, sounds like docValues is set. >> >> Sure, in solrconfig.xml, there are two sections “firstSearcher” and >> “newSearcher”. >> These are queries (or lists of queries) that are fired as part of >> autowarming >> when Solr is first started (firstSearcher) or when a commit happens that >> opens >> a new searcher (newSearcher). These are hand-crafted static queries. So >> create one or more newSearcher sections in that block that exercise your >> faceting and it’ll be fired as part of autowarming. That should smooth out >> the delay your user’s experience when commits happen. >> >> Best, >> Erick >> >>> On Mar 30, 2020, at 4:06 PM, Revas <revas2...@gmail.com> wrote: >>> >>> Thanks, Eric. >>> >>> 1) We are using dynamic string field for faceting where indexing =false >> and >>> stored=false . By default docValues are enabled for primitive fields >> (solr >>> 6.6.), so not explicitly defined in schema. Do you think its wrong >>> assumption? Also I do not this field listed in feild cache, but dont see >>> any dynamic fields listed. >>> 2) Autowarm count is at 32 for both and autowarm time is 25 for >> queryresult >>> and 17 >>> 3)Can you elaborate what you mean here >>> >>> >>> >>> On Mon, Mar 30, 2020 at 1:43 PM Erick Erickson <erickerick...@gmail.com> >>> wrote: >>> >>>> Response spikes after commits are almost always something to do >>>> with autowarming or docValues being set to false. So here’s what >>>> I’d look at, in order. >>>> >>>> 1> are the fields used defined with docValues=true? They should be. >>>> With this much variance it sounds like you don’t have that value set. >>>> You’ll have to rebuild your entire index, first deleting all documents… >>>> >>>> You assert that they are all docValues, but the variance is so >>>> high that I wonder whether they _all_ are. They may very well be, but >>>> I’ve been tripped up by things I know are true that aren’t too often ;) >>>> >>>> You can insure this by setting 'uninvertible=“true” ‘ in your field >> type, >>>> see: https://issues.apache.org/jira/browse/SOLR-12962 if you’re on >>>> 7.6 or later. >>>> >>>> 2>what are your autowarming settings for queryResultCache and/or >>>> filterCache. Start with a relatively small number, say 16 and look at >>>> your autowarm times to insure they aren’t excessive. >>>> >>>> 3> if autowarming doesn’t help, consider specifying a newSearcher >>>> event in solrconfig.xml that exercises the facets. >>>> >>>> NOTE: <2> and <3> will mask any fields that are docValues=false that >>>> slipped through the cracks, so I’d double check <1> first. >>>> >>>> Best, >>>> Erick >>>> >>>>> On Mar 30, 2020, at 12:20 PM, sujatha arun <revas2...@gmail.com> >> wrote: >>>>> >>>>> A facet heavy query which uses docValue fields for faceting returns >>>> about >>>>> 5k results executes between 10ms to 5 secs and the 5 secs time seems >> to >>>>> coincide with after a hard commit. >>>>> >>>>> Does that have any relation? Why the fluctuation in execution time? >>>>> >>>>> Thanks, >>>>> Revas >>>> >>>> >> >>
