Excellent reply.
You mentioned: "I've been experimenting with FastLRUCache versus
LRUCache, because I read that below a certain hitratio, the latter is
better."
Do you happen to remember what that threshold is? Thanks
On 12/14/10 7:59 AM, Shawn Heisey wrote:
On 12/14/2010 8:31 AM, Mark wrote:
Can anyone offer some advice on what some good settings would be for
an index or around 6 million documents totaling around 20-25gb? It
seems like when our index gets to this size our CPU load spikes
tremendously.
If you are adding, deleting, or updating documents on a regular basis,
I would bet that it's your autoWarmCount. You've told it that
whenever you do a commit, it needs to make up to 32768 queries against
the new index. That's very intense and time-consuming. If you are
also optimizing the index, the problem gets even worse. On the
documentCache, autowarm doesn't happen, so the 16384 specified there
isn't actually doing anything.
Below are my settings. I originally had much larger caches with
equally large autoWarmCounts ... reducing them to this level was the
only way I could get my autowarm time below 30 seconds on each index.
If you go to the admin page for your index and click on Statistics,
then search for "warmupTime" you'll see how long it took to do the
queries. Later on the page you'll also see this broken down on each
cache.
Since I made the changes, performance is actually better now, not
worse. I have been experimenting with FastLRUCache versus LRUCache,
because I read that below a certain hitratio, the latter is better.
I've got 8 million documents in each shard, taking up about 15GB.
My mergeFactor is 16 and my ramBufferSize is 256MB. These really only
come into play when I do a full re-index, which is rare.
<filterCache
class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="32"
cleanupThread="true"
/>
<queryResultCache
class="solr.FastLRUCache"
size="1024"
initialSize="1024"
autowarmCount="64"
cleanupThread="true"
/>
<documentCache
class="solr.LRUCache"
size="16384"
initialSize="4096"
cleanupThread="true"
/>
