I reindexed and optimized and it helped. However now each query averages
about 1 second(down from 3-4 seconds). The bottleneck now is the
getFacetTermEnumCounts function. If I take that call out it is a non
measurable query time and the filtercache is being used. With the
getFacetTermEnumCounts in, the filter cache after three queries is below
with the hitration at 0 and everything is being evicted. This call is for
the brand/manufacturer so I'm sure it is going through many thousands of
queries. I'm thinking about pre-processing the brand/manu to get a small set
of top brands per category and just quering them no matter what the other
facets are set to.(with certain filters, no brands will be shown) If I
still want to call the getFacetTermEnumCounts for ALL brands, why is it not
using the cache?
lookups : 32849
hits : 0
hitratio : 0.00
inserts : 32850
evictions : 32338
size : 512
cumulative_lookups : 32849
cumulative_hits : 0
cumulative_hitratio : 0.00
cumulative_inserts : 32850
cumulative_evictions : 32338
Thanks,
Mike
----- Original Message -----
From: "Yonik Seeley" <[EMAIL PROTECTED]>
To: <solr-user@lucene.apache.org>
Sent: Tuesday, December 05, 2006 8:46 PM
Subject: Re: Performance issue.
On 12/5/06, Gmail Account <[EMAIL PROTECTED]> wrote:
> There's nothing wrong with CPU jumping to 100% each query, that just
> means you aren't IO bound :-)
What do you mean not IO bound?
There is always going to be a bottleneck somewhere. In very large
indicies, the bottleneck may be waiting for IO (waiting for data to be
read from the disk). If you are on a single processor system and you
aren't waiting for data to be read from the disk or the network, then
the request will be using close to 100% CPU, which is actually a good
thing.
The bad thing is how long the query takes, not the fact that it's CPU
bound.
>> > - I did an optimize index through Luke with compound format and
>> > noticed
>> > in the solrconfig file that useCompoundFile is set to false.
>
> Don't do this unless you really know what you are doing... Luke is
> probably using a different version of Lucene than Solr, and it could
> be dangerous.
Do you think I should reindex everything?
That would be the safest thing to do.
> - if you are using filters, any larger than 3000 will be double the
> size (maxDoc bits)
What do you mean larger than 3000? 3000 what and how do I tell?
From solrconfig.xml:
<!-- This entry enables an int hash representation for filters
(DocSets)
when the number of items in the set is less than maxSize. For
smaller
sets, this representation is more memory efficient, more efficient
to
iterate over, and faster to take intersections. -->
<HashDocSet maxSize="3000" loadFactor="0.75"/>
The key is that the memory consumed by a HashDocSet is independent of
maxDoc (the maximum internal lucene docid), but a BitSet based set has
maxDoc bits in it. Thus, an unoptimized index with more deleted
documents causes a higher maxDoc and higher memory usage for any
BitSet based filters.
-Yonik
