Hello Steve, debugQuery=true shows whether it's facets or query, whether it's query parsing or searching (prepare vs process), cache statistics can tell about its' efficiency; sometimes a problem is obvious from request parameters. Simple sampling with jconsole or even by jstack can point on a smoking gun.
On Fri, Nov 20, 2015 at 4:08 PM, Steven White <swhite4...@gmail.com> wrote: > Thanks Erick. > > The 1500 fields is a design that I inherited. I'm trying to figure out why > it was done as such and what it will take to fix it. > > What about my other question: how does one go about debugging performance > issues in Solr to find out where time is mostly spent? How do I know my > Solr parameters, such as cache and what have you are set right? From what > I see, we are using the defaults off solrconfig.xml. > > I'm on Solr 5.2 > > Steve > > > On Thu, Nov 19, 2015 at 11:36 PM, Erick Erickson <erickerick...@gmail.com> > wrote: > > > An fq is still a single entry in your filterCache so from that > > perspective it's the same. > > > > And to create that entry, you're still using all the underlying fields > > to search, so they have to be loaded just like they would be in a q > > clause. > > > > But really, the fundamental question here is why your design even has > > 1,500 fields and, more specifically, why you would want to search them > > all at once. From a 10,000 ft. view, that's a very suspect design. > > > > Best, > > Erick > > > > On Thu, Nov 19, 2015 at 4:06 PM, Walter Underwood <wun...@wunderwood.org > > > > wrote: > > > The implementation for fq has changed from 4.x to 5.x, so I’ll let > > someone else answer that in detail. > > > > > > In 4.x, the result of each filter query can be cached. After that, they > > are quite fast. > > > > > > wunder > > > Walter Underwood > > > wun...@wunderwood.org > > > http://observer.wunderwood.org/ (my blog) > > > > > > > > >> On Nov 19, 2015, at 3:59 PM, Steven White <swhite4...@gmail.com> > wrote: > > >> > > >> Thanks Walter. I see your point. Does this apply to fq as will? > > >> > > >> Also, how does one go about debugging performance issues in Solr to > find > > >> out where time is mostly spent? > > >> > > >> Steve > > >> > > >> On Thu, Nov 19, 2015 at 6:54 PM, Walter Underwood < > > wun...@wunderwood.org> > > >> wrote: > > >> > > >>> With one field in qf for a single-term query, Solr is fetching one > > posting > > >>> list. With 1500 fields, it is fetching 1500 posting lists. It could > > easily > > >>> be 1500 times slower. > > >>> > > >>> It might be even slower than that, because we can’t guarantee that: > a) > > >>> every algorithm in Solr is linear, b) that all those lists will fit > in > > >>> memory. > > >>> > > >>> wunder > > >>> Walter Underwood > > >>> wun...@wunderwood.org > > >>> http://observer.wunderwood.org/ (my blog) > > >>> > > >>> > > >>>> On Nov 19, 2015, at 3:46 PM, Steven White <swhite4...@gmail.com> > > wrote: > > >>>> > > >>>> Hi everyone > > >>>> > > >>>> What is considered too many fields for qf and fq? On average I will > > have > > >>>> 1500 fields in qf and 100 in fq (all of which are OR'ed). Assuming > I > > can > > >>>> (I have to check with the design) for qf, if I cut it down to 1 > field, > > >>> will > > >>>> I see noticeable performance improvement? It will take a lot of > > effort > > >>> to > > >>>> test this which is why I'm asking first. > > >>>> > > >>>> As is, I'm seeing 2-5 sec response time for searches on an index of > 1 > > >>>> million records with total index size (on disk) of 4 GB. I gave > Solr > > 2 > > >>> GB > > >>>> of RAM (also tested at 4 GB) in both cases Solr didn't use more > then 1 > > >>> GB. > > >>>> > > >>>> Thanks in advanced > > >>>> > > >>>> Steve > > >>> > > >>> > > > > > > -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics <http://www.griddynamics.com> <mkhlud...@griddynamics.com>
