couple of random questions:
1) in some of your example queries (like query 2) there
trimTime:[2012-09-04T15:23:48Z
TO *] mentioned in two diff. filter queries of the same request. Any reason
to do this?
2) if you don't need the documents themselves, only the facets, you could
set rows=0.
3) try tweaking cache parameters, e.g. have a look at filterCache. Stats
page of solr admin should help gain an insight of cache utilization in your
setup.
Dmitry
On Wed, Oct 10, 2012 at 3:38 PM, Yuval Dotan <yuvaldo...@gmail.com> wrote:
> OK so I solved the question about the query that returns no results and
> still takes time - I needed to add the facet.mincount=1 parameter and this
> reduced the time to 200-300 ms instead of seconds.
>
> I still could't figure out why a query that returns very few results (like
> query number 2) still takes seconds to return even with
> the facet.mincount=1 parameter.
> I couldn't understand why the facet pivot takes so much time on 299 docs.
>
> Does anyone have any idea?
>
> Example Query:
> ----------------------------------------------------
> (2)
> q=*:*&fq=(trimTime:[2012-09-04T15:23:48Z TO *])&fq=(Severity:("High"
> "Critical"))&fq=(trimTime:[2012-09-04T15:23:48Z TO
> *])&fq=(Confidence_Level:("N/A")) OR (Confidence_Level:("Medium-High")) OR
>
> (Confidence_Level:("High"))&f.product.facet.sort=index&f.product.facet.limit=-1&f.Severity.facet.sort=index&f.Severity.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
>
> NumFound: 299
>
> Times(ms):
> Qtime: 2,756 Query: 307 Facet: 2,449
>
> On Thu, Sep 20, 2012 at 5:24 PM, Yuval Dotan <yuvaldo...@gmail.com> wrote:
>
> > Hi,
> >
> > We have a system that inserts logs continuously (real-time).
> > We have been using the Solr facet pivot feature for querying and have
> been
> > experiencing slow query times and we were hoping to gain some insights
> with
> > your help.
> > schema and solrconfig are attached
> >
> > Here are our questions (data below):
> >
> > 1. Why is facet time so long in (3) and (5) - in cases where there are
> > 0 or very few results?
> > 2. We ran two queries that are only differ in the time limit (for the
> > second query - time range is very small) - we got the same time for
> both
> > queries although the second one returned very few results - again why
> is
> > that?
> > 3. Is there a way to improve pivot facet time?
> >
> > System Data:
> >
> > Index size: 63 GB
> > RAM:4Gb
> > CPU: 2 x Xeon E5410 2.33GHz
> > Num of Documents: 109,278,476
> >
> >
> > query examples:
> >
> >
> ---------------------------------------------------------------------------------------------------------------------------------------------
> > (1)
> > Query:
> > q=*:*&fq=(trimTime:[2012-09-04T14:29:24Z TO
> > *])&fq=(trimTime:[2012-09-04T14:29:24Z TO
> >
> *])&f.product.facet.sort=index&f.product.facet.limit=-1&f.Severity.facet.sort=index&f.Severity.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
> >
> > NumFound:
> > 11,407,889
> >
> > Times (ms):
> > Qtime: 3,239 Query: 353 Facet: 2,885
> >
> ---------------------------------------------------------------------------------------------------------------------------------------------
> >
> > (2)
> > Query:
> > q=*:*&fq=(trimTime:[2012-09-04T15:23:48Z TO *])&fq=(Severity:("High"
> > "Critical"))&fq=(trimTime:[2012-09-04T15:23:48Z TO
> > *])&fq=(Confidence_Level:("N/A")) OR (Confidence_Level:("Medium-High"))
> OR
> >
> (Confidence_Level:("High"))&f.product.facet.sort=index&f.product.facet.limit=-1&f.Severity.facet.sort=index&f.Severity.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
> >
> > NumFound: 299
> >
> > Times(ms):
> > Qtime: 2,756 Query: 307 Facet: 2,449
> >
> >
> ---------------------------------------------------------------------------------------------------------------------------------------------
> > (3)
> > Query:
> > q=*:*&fq=(trimTime:[2012-09-11T12:55:00Z TO *])&fq=(Severity:("High"
> > "Critical"))&fq=(trimTime:[2012-09-04T15:23:48Z TO
> > *])&fq=(Confidence_Level:("N/A")) OR (Confidence_Level:("Medium-High"))
> OR
> >
> (Confidence_Level:("High"))&f.product.facet.sort=index&f.product.facet.limit=-1&f.Severity.facet.sort=index&f.Severity.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
> >
> > NumFound: 7
> >
> > Times(ms):
> > Qtime: 2,798 Query: 312 Facet: 2,485
> >
> >
> ---------------------------------------------------------------------------------------------------------------------------------------------
> > (4)
> > Query:
> > q=*:*&fq=(trimTime:[2012-09-04T15:43:16Z TO
> > *])&fq=(trimTime:[2012-09-04T15:43:16Z TO *])&fq=(product:("Application
> > Control")) OR (product:("URL
> >
> Filtering"))&f.appi_name.facet.sort=index&f.appi_name.facet.limit=-1&f.app_risk.facet.sort=index&f.app_risk.facet.limit=-1&f.matched_category.facet.sort=index&f.matched_category.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.appi_name.facet.method=enum&facet.pivot=appi_name,app_risk,matched_category,trimTimeex&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
> >
> > NumFound: more than 30M
> >
> > Times(ms): Qtime: 23,288
> >
> ---------------------------------------------------------------------------------------------------------------------------------------------
> >
> > (5)
> > Query:
> > q=*:*&fq=(trimTime:[2012-09-05T06:03:55Z TO *])&fq=(Severity:("High"
> > "Critical"))&fq=(trimTime:[2012-09-05T06:03:55Z TO
> *])&fq=(product:("IPS"))
> > OR (product:("SmartDefense"))&fq=(action:("Detect")) OR
> > (action:("mixed"))&fq=(Confidence_Level:("Medium-High")) OR
> >
> (Confidence_Level:("High"))&f.product.facet.sort=index&f.product.facet.limit=-1&f.Severity.facet.sort=index&f.Severity.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
> >
> > NumFound: 0
> >
> > Times(ms): Qtime: 2,971 Query: 218 Facet: 2,753
> >
> >
> ---------------------------------------------------------------------------------------------------------------------------------------------
> >
>
