Hi Jay, Please separate the clauses. Feed one of them to the main q parameter with content score operator =^ since you are sorting on a structured field(e.g. date)
q:fieldB:(123 OR 456)^=1.0 &fq=dt1:[date1 TO *] &fq=dt2:[* TO NOW/DAY+1] &fq=fieldA:abc &sort=dt1 asc,field2 asc, fieldC desc Play with the caches. Also consider disabling caching, and/or supplying execution order for the filer queries. Please see : https://lucidworks.com/blog/2012/02/10/advanced-filter-caching-in-solr/ Ahmet On Friday, May 27, 2016 4:01 PM, Jay Potharaju <jspothar...@gmail.com> wrote: I updated almost 1/3 of the data and ran my queries with new columns as mentioned earlier. The query returns data in almost half the time as compared to before. I am thinking that if I update all the columns there would not be much difference in query response time. <field name="dt1" type="date" indexed="true" stored="true"/> <field name=" dt1_new" type="tdate" indexed="true" stored="true"/> <field name="dt2" type="date" indexed="true" stored="true"/> <field name=" dt2_new" type="tdate" indexed="true" stored="true" docValues="true" default=""/> Are there any suggestions on how handle filtering/querying/sorting on high cardinality date fields? Index size: 30Million Solr: 4.3.1 Thanks On Thu, May 26, 2016 at 6:04 AM, Jay Potharaju <jspothar...@gmail.com> wrote: > Hi, > Thanks for the feedback. The queries I run are very basic filter queries > with some sorting. > > q:*:*&fq=(dt1:[date1 TO *] && dt2:[* TO NOW/DAY+1]) && fieldA:abc && > fieldB:(123 OR 456)&sort=dt1 asc,field2 asc, fieldC desc > > I noticed that the date fields(dt1,dt2) are using date instead of tdate > fields & there are no docValues set on any of the fields used for sorting. > > In order to fix this I plan to add a new field using tdate & docvalues > where required to the schema & update the new columns only for documents > that have fieldA set to abc. Once the fields are updated query on the new > fields to measure query performance . > > > - Would the new added fields be used effectively by the solr index > when querying & filtering? What I am not sure is whether only populating > small number of documents(fieldA:abc) that are used for the above query > provide performance benefits. > - Would there be a performance penalty because majority of the > documents(!fieldA:abc) dont have values in the new columns? > > Thanks > > On Wed, May 25, 2016 at 8:40 PM, Jay Potharaju <jspothar...@gmail.com> > wrote: > >> Any links that illustrate and talk about solr internals and how >> indexing/querying works would be a great help. >> Thanks >> Jay >> >> On Wed, May 25, 2016 at 6:30 PM, Jay Potharaju <jspothar...@gmail.com> >> wrote: >> >>> Hi, >>> Thanks for the feedback. The queries I run are very basic filter queries >>> with some sorting. >>> >>> q:*:*&fq=(dt1:[date1 TO *] && dt2:[* TO NOW/DAY+1]) && fieldA:abc && >>> fieldB:(123 OR 456)&sort=dt1 asc,field2 asc, fieldC desc >>> >>> I noticed that the date fields(dt1,dt2) are using date instead of tdate >>> fields & there are no docValues set on any of the fields used for sorting. >>> >>> In order to fix this I plan to add a new field using tdate & docvalues >>> where required to the schema & update the new columns only for documents >>> that have fieldA set to abc. Once the fields are updated query on the new >>> fields to measure query performance . >>> >>> >>> - Would the new added fields be used effectively by the solr index >>> when querying & filtering? What I am not sure is whether only populating >>> small number of documents(fieldA:abc) that are used for the above query >>> provide performance benefits. >>> - Would there be a performance penalty because majority of the >>> documents(!fieldA:abc) dont have values in the new columns? >>> >>> >>> Thanks >>> Jay >>> >>> On Tue, May 24, 2016 at 8:06 PM, Erick Erickson <erickerick...@gmail.com >>> > wrote: >>> >>>> Try adding debug=timing, that'll give you an idea of what component is >>>> taking all the time. >>>> From there, it's "more art than science". >>>> >>>> But you haven't given us much to go on. What is the query? Are you >>>> grouping? >>>> Faceting on high-cardinality fields? Returning 10,000 rows? >>>> >>>> Best, >>>> Erick >>>> >>>> On Tue, May 24, 2016 at 4:52 PM, Ahmet Arslan <iori...@yahoo.com.invalid> >>>> wrote: >>>> > >>>> > >>>> > Hi, >>>> > >>>> > Is it QueryComponent taking time? >>>> > Ot other components? >>>> > >>>> > Also make sure there is plenty of RAM for OS cache. >>>> > >>>> > Ahmet >>>> > >>>> > On Wednesday, May 25, 2016 1:47 AM, Jay Potharaju < >>>> jspothar...@gmail.com> wrote: >>>> > >>>> > >>>> > >>>> > Hi, >>>> > I am trying to debug solr performance problems on an old version of >>>> solr, >>>> > 4.3.1. >>>> > The queries are taking really long -in the range of 2-5 seconds!!. >>>> > Running filter query with only one condition also takes about a >>>> second. >>>> > >>>> > There is memory available on the box for solr to use. I have been >>>> looking >>>> > at the following link but was looking for some more reference that >>>> would >>>> > tell me why a particular query is slow. >>>> > >>>> > https://wiki.apache.org/solr/SolrPerformanceProblems >>>> > >>>> > Solr version:4.3.1 >>>> > Index size:128 GB >>>> > Heap:65 GB >>>> > Index size:75 GB >>>> > Memory usage:70 GB >>>> > >>>> > Even though there is available memory is high all is not being used >>>> ..i >>>> > would expect the complete index to be in memory but it doesnt look >>>> like it >>>> > is. Any recommendations ?? >>>> > >>>> > -- >>>> > Thanks >>>> > Jay >>>> >>> >>> >>> >>> -- >>> Thanks >>> Jay Potharaju >>> >>> >> >> >> >> -- >> Thanks >> Jay Potharaju >> >> > > > > -- > Thanks > Jay Potharaju > > -- Thanks Jay Potharaju
