Hi,
The criteria rarely varies from others that are much quicker, maybe only
what the start row is. Most of the time the main "terms" are a single word
or just a "blank query" (q.alt=*:*)
My request handler does have a lot of predefined filters, this is included
below. Most of this is auto-warmed.
The server also does updates via the DataImportHandler every 5 minutes.
Optimisation is only performed once a day at approximately midnight. These
high response times can happen at any time of day, mostly out of working
hours, which is also when we have the least number of updates + search
traffic.
In terms of CPU and IO usage, as mentioned above they are mostly out of
hours so I will see if our server admins have setup some SNMP tools to
provide reports for me. Looking at the server right now I can see between 2
and 10% CPU
Here is a small extract from the log:
21-Jan-2009 19:45:39 org.apache.solr.core.SolrCore execute
INFO: [news] webapp=/solr path=/select
params={rows=10&start=40&sort=score+desc,+newsArticleDate_Date+desc,+newsCalculatedImportance+desc&fq=newsArticleDate_Year:1995&hl=true&qt=BR2News}
hits=1106 status=0 QTime=31
21-Jan-2009 19:45:39 org.apache.solr.core.SolrCore execute
INFO: [news] webapp=/solr path=/select
params={rows=10&start=100&sort=score+desc,+newsArticleDate_Date+desc,+newsCalculatedImportance+desc&fq=newsArticleDate_Year:1996&hl=true&qt=BR2News}
hits=8345 status=0 QTime=119234
The request handler is:
<requestHandler name="BR2News" class="solr.DisMaxRequestHandler" >
<lst name="defaults">
<int name="rows">10</int>
<str name="hl">false</str>
<str name="sort">score desc, newsArticleDate_Date desc,
newsCalculatedImportance desc</str>
<str name="f.newsArticleDate_Year.facet.sort">false</str>
<str name="f.newsArticleDate_Month.facet.sort">false</str>
<str name="f.newsArticleDate_Day.facet.sort">false</str>
<str name="facet.mincount">1</str>
<str name="mlt">false</str>
<str name="wt">xslt</str>
<str name="tr">newsResults.xsl</str>
</lst>
<lst name="appends">
<str name="fq">news_magJournalCode:BR2 OR news_magJournalCode:CAM OR
news_magJournalCode:CEI OR news_magJournalCode:CIT OR
news_magJournalCode:DRN OR news_magJournalCode:EVE OR
news_magJournalCode:MKT OR news_magJournalCode:MXD OR
news_magJournalCode:PRA OR news_magJournalCode:PRI OR
news_magJournalCode:PRS OR news_magJournalCode:PRW OR
news_magJournalCode:REV OR news_magJournalCode:RSV OR
news_magJournalCode:WWP OR news_magJournalCode:XMB OR
news_magJournalCode:XMW OR news_magJournalCode:XX6</str>
<str name="fq">newsStatus:true</str>
<str name="fq">newsPublishedDate_Date:[* TO NOW/DAY]</str>
<str name="fq">newsArticleDate_Date:[* TO NOW/DAY]</str>
</lst>
<lst name="invariants">
<!-- mm=1 ONLY TOUCH THIS IF YOU REALLY KNOW WHAT YOU ARE DOING!!!!!
-->
<str name="mm">1</str>
<!-- mm=1 ONLY TOUCH THIS IF YOU REALLY KNOW WHAT YOU ARE DOING!!!!!
-->
<str name="hl.simple.pre"><![CDATA[<span class="hiLite">]]></str>
<str name="hl.simple.post"><![CDATA[</span>]]></str>
<str name="f.newsBody.hl.snippets">3</str>
<str name="f.newsBody.hl.mergeContiguous">true</str>
<str name="q.alt">*:*</str>
<str name="echoParams">all</str>
<float name="tie">0.01</float>
<str
name="fl">newsID,newsTitle,newsTitleAlternate,newsAuthor,news_magJournalCode,news_newsTypeID,newsStatus,magName,newsSeoURLTitle,newsSummary,newsBody,newsSummaryAlternate,newsArticleDate_DateTime,newsDateAdded_DateTime,newsAuthor,score</str>
<str name="version">2.2</str>
<str name="qf">newsTitle newsSummary^0.75 newsBody^0.5 newsAuthor^0.1</str>
<!-- no "exact" author field as this is already indexed appropriately -->
<str name="pf">newsTitleExact newsSummaryExact newsBodyExact
newsAuthor</str>
<str name="ps">1</str>
<str name="hl.fl">newsTitle newsBody newsSummary newsAuthor</str>
<str name="mlt.fl">newsTitle newsBody newsSummary newsAuthor</str>
<str name="facet">true</str>
<str name="facet.field">news_magJournalCode_FacetDetails</str>
<str name="facet.field">news_newsTypeID_FacetDetails</str>
<str name="facet.field">sector_FacetDetails</str>
<str name="facet.field">discipline_FacetDetails</str>
<str name="facet.field">asset_FacetDetails</str>
<!-- get facet for "today" -->
<str name="facet.query">newsArticleDate_Date:[NOW/DAY TO NOW/DAY]</str>
<!-- get facet for "lastweek" (last 7 days) -->
<str name="facet.query">newsArticleDate_Date:[NOW/DAY-7DAYS TO
NOW/DAY]</str>
<!-- get facet for "lastmonth" -->
<str name="facet.query">newsArticleDate_Date:[NOW/DAY-1MONTH TO
NOW/DAY]</str>
<str name="facet.field">newsArticleDate_Year</str>
<str name="facet.field">newsArticleDate_Month</str>
<str name="facet.field">newsArticleDate_Day</str>
<!-- get facet for "today" -->
<str name="facet.query">newsDateAdded_Date:[NOW/DAY TO NOW/DAY]</str>
<!-- get facet for "lastweek" (last 7 days) -->
<str name="facet.query">newsDateAdded_Date:[NOW/DAY-7DAYS TO NOW/DAY]</str>
<!-- get facet for "lastmonth" -->
<str name="facet.query">newsDateAdded_Date:[NOW/DAY-1MONTH TO NOW/DAY]</str>
<str name="facet.field">newsDateAdded_Year</str>
<str name="facet.field">newsDateAdded_Month</str>
<str name="facet.field">newsDateAdded_Day</str>
</lst>
</requestHandler>
On Thu, Jan 22, 2009 at 2:23 PM, Otis Gospodnetic <
otis_gospodne...@yahoo.com> wrote:
> Hi,
>
> Is there anything special about those queries? e.g. lots of terms,
> frequent terms, something else? Is there anything else happening on that
> server when you see such long queries? Do you see lots of IO or lots of CPU
> being used during those times?
>
>
> Otis --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
> > From: hbi dev <hbi...@googlemail.com>
> > To: solr-user@lucene.apache.org
> > Sent: Thursday, January 22, 2009 6:39:39 AM
> > Subject: Intermittent high response times
> >
> > Hi all,
> > I have an implmentation of solr (rev.708837) running on tomcat 6.
> >
> > Approx 600,000 docs, 2 fairly content heavy text fields, between 4 and 7
> > facets (depending on what our front end is requesting, and mostly low
> unique
> > values)
> >
> > 1GB of memory allocated, generally I do not see it using all of that up.
> >
> > For the most part my response times are under 200ms, but I randomly get
> > times that are around 100,000ms!
> >
> > Original load testing didn't reveal this, I can see from the logs we are
> > getting approx 20 requests per second so it's not really under much load
> at
> > the moment.
> >
> > Does anyone have any pointers that I can follow or look into?
> > Please ask if I need to provide any more info.
> >
> > Thanks in advance
> >
> > Regards,
> > Waseem
>
>
