Well, as Eric said, you're apparently calculating
the scores on over 3M documents since you're
querying on very common words ('if', 'a', etc).
Have you tried simply using a stopword list?
Best
Erick
On Fri, Aug 17, 2012 at 10:53 AM, Sujatha Arun <suja.a...@gmail.com> wrote:
> Hi,
>
> This is the parsed query string which *returns 3331234 documents*
>
> <str name="parsedquery_toString">content:elena content:read content:a
> content:rate content:r content:page content:per content:minut content:a
> content:total content:m content:minut</str> re
>
> and a Boolean "AND " query of the above query takes 410 ms and *returns
> just 1 result*
>
> <str name="parsedquery_toString">+content:elena +content:read +content:a
> +content:rate +content:r +content:page +content:per +content:minut
> +content:a +content:total +content:m +content:minut</str>
>
> so the main difference being the number of results returned ,so is this
> i/0 issue or is this time expected given the number of documents returned.
>
> Also what is the difference between Qtime and timing shown as a part of
> Debugquery?
>
> Regards
> Sujatha
>
> Regards
> Sujatha
>
>
>
> On Fri, Aug 17, 2012 at 8:35 PM, Jack Krupansky
> <j...@basetechnology.com>wrote:
>
>> First, provide us with the additional info Erik requested in his last
>> reply.
>>
>> Then, can you provide a snippet from your Solr log file that shows a
>> couple of queries. Maybe there is something else going on or some
>> exceptions, or something. Or at least to show us the quey times in context.
>>
>> Finally, has this been happening from the very beginning, or did it begin
>> to grow gradually and steadily, or did it just suddenly start to happen
>> even though it had been fine just a few minutes earlier? If the latter, had
>> there been and system or JVM or Solr configuration changes between the time
>> queries were fast and when they became slow?
>>
>> Thanks.
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: Sujatha Arun
>> Sent: Friday, August 17, 2012 10:50 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Muticore Sharding
>>
>>
>> Sorry typo ,I meant upwards of 20s at any time. What should I be looking
>> at?
>>
>> Regards
>> Sujatha
>>
>> On Fri, Aug 17, 2012 at 8:09 PM, Sujatha Arun <suja.a...@gmail.com> wrote:
>>
>> Erik,
>>>
>>> What could be the issue Load / I/O ? It seems to shows upwards of 20 ms
>>> at any time
>>>
>>> Regards
>>> Sujatha
>>>
>>>
>>> On Fri, Aug 17, 2012 at 6:18 PM, Erik Hatcher <erik.hatc...@gmail.com
>>> >wrote:
>>>
>>> Sujatha - that query debug output shows only 218ms, so it isn't
>>>> representative of the issue you're reporting.
>>>>
>>>> Also, what's the query parse output? I imagine you're doing a boolean
>>>> OR query across all those terms (include "a"), yes? Maybe you'd rather
>>>> the
>>>> operator be AND?
>>>>
>>>> Erik
>>>>
>>>>
>>>> On Aug 17, 2012, at 08:01 , Sujatha Arun wrote:
>>>>
>>>> > No customization,its the default standard request handler.Solr Version
>>>> is
>>>> > 1.3
>>>> >
>>>> > "a" is not there in stop words
>>>> >
>>>> > Server Load ,i presume is not there , but not too sure ,not checked.
>>>> >
>>>> > RAM :
>>>> >
>>>> > TOTAL RAM :48GB
>>>> > RAM to JVM :18 GB ,Permgen =2GB
>>>> > TOTAL INDEX SIZE of all the multicore Instances =23GB
>>>> >
>>>> > Timing [Cut & paste ] , I have never looked at this before
>>>> >
>>>> > <str name="QParser">**OldLuceneQParser</str><lst name="timing"><double
>>>> > name="time">218.0</double><lst name="prepare"><double
>>>> > name="time">6.0</double><lst
>>>> > name="org.apache.solr.handler.**component.QueryComponent"><**double
>>>> > name="time">5.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.FacetComponent"><**double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.**
>>>> MoreLikeThisComponent"><double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.HighlightComponent">**
>>>> <double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.DebugComponent"><**double
>>>> > name="time">0.0</double></lst>**</lst><lst name="process"><double
>>>> > name="time">211.0</double><lst
>>>> > name="org.apache.solr.handler.**component.QueryComponent"><**double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.FacetComponent"><**double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.**
>>>> MoreLikeThisComponent"><double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.HighlightComponent">**
>>>> <double
>>>> > name="time">0.0</double></lst>**<lst
>>>> > name="org.apache.solr.handler.**component.DebugComponent"><**double
>>>> > name="time">211.0</double></**lst></lst></lst></lst>
>>>> >
>>>> > Regards
>>>> > Sujatha
>>>> >
>>>> > On Fri, Aug 17, 2012 at 4:11 PM, Erik Hatcher <erik.hatc...@gmail.com
>>>> >wrote:
>>>> >
>>>> >> Just over 4M docs... no need to shard.
>>>> >>
>>>> >> Both your queries contain what is likely very common terms (content:a
>>>> in
>>>> >> the first one and content:1 in the second). Generally these are "stop
>>>> >> words" and removed either during indexing or querying, but I guess not
>>>> in
>>>> >> your case.
>>>> >>
>>>> >> What's your "standard" request handler look like? Doing anything
>>>> custom
>>>> >> in there?
>>>> >>
>>>> >> What are the timings of the components in the debugQuery=true >>
>>>> response?
>>>> >>
>>>> >> Is the server under load when you're issuing these queries? What
>>>> about
>>>> >> RAM?
>>>> >>
>>>> >> Again, what version of Solr?
>>>> >>
>>>> >> Erik
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Aug 17, 2012, at 06:35 , Sujatha Arun wrote:
>>>> >>
>>>> >>> Hi Erick,
>>>> >>>
>>>> >>> The number of documents is : 4389048
>>>> >>>
>>>> >>> I have given 2 queries below with timing and the number of hits
>>>> >>>
>>>> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>>>> >>>
>>>> >>
>>>> params={version=2.1&fl=*+**score&stylesheet=&qt=standard&**
>>>> fq=&rows=100&start=0&q=**content:if+content:elena+**
>>>> content:reads+content:at+**content:a+content:rate+**
>>>> content:of+content:r+content:**pages+content:per+content:**
>>>> minute+content:for+content:a+**content:total+content:of+**
>>>> content:m+content:minutes}
>>>> >>> hits=3331109 status=0 QTime=22677
>>>> >>>
>>>> >>> INFO: [ipc_widget_search] webapp=/multicore_5 path=/select/
>>>> >>>
>>>> >>
>>>> params={version=2.1&fl=*+**score&stylesheet=&qt=standard&**
>>>> fq=&rows=100&start=0&q=**content:chapter+content:1}
>>>> >>> hits=1919445 status=0 QTime=6677
>>>> >>>
>>>> >>>
>>>> >>> Any time I execute the time taken to execute the first query is >
>>>> than
>>>> >>> 22secs ,now it took 33 secs to execute.
>>>> >>>
>>>> >>> Regards
>>>> >>> Sujatha
>>>> >>>
>>>> >>>
>>>> >>> On Fri, Aug 17, 2012 at 3:30 PM, Erik Hatcher <
>>>> erik.hatc...@gmail.com
>>>> >>> wrote:
>>>> >>>
>>>> >>>> How many documents do you have? What are the queries? I'd guess
>>>> your
>>>> >>>> query complexity (or load) is to blame here, not index size. What
>>>> >> version
>>>> >>>> of Solr?
>>>> >>>>
>>>> >>>> Until you know what is causing the slow queries, sharding is not
>>>> >> something
>>>> >>>> to consider I'd say. But yes, you would want to reindex to
>>>> distribute
>>>> >> the
>>>> >>>> documents.
>>>> >>>>
>>>> >>>> Erik
>>>> >>>>
>>>> >>>>
>>>> >>>> On Aug 17, 2012, at 02:21 , Sujatha Arun wrote:
>>>> >>>>
>>>> >>>>> Hello,
>>>> >>>>>
>>>> >>>>> One of the Index in a multicore set up has a 3GB+ index ,and it
>>>> seems
>>>> >> to
>>>> >>>>> take around 5000ms+ to return simple boolean queries . The Index is
>>>> >>>>> not optimized
>>>> >>>>> Would it make sense to shard the index as cores in the same server
>>>> to
>>>> >>>>> expect better response time?
>>>> >>>>>
>>>> >>>>> do I have to re index all over again to distribute the documents
>>>> >> between
>>>> >>>> 2
>>>> >>>>> shards ?
>>>> >>>>>
>>>> >>>>> Regards
>>>> >>>>> Sujatha
>>>> >>>>
>>>> >>>>
>>>> >>
>>>> >>
>>>>
>>>>
>>>>
>>>
>>
