Another trick is to read in the parts of the index file that you
search against: term dictionary and maybe a few others. (The Lucene
wiki describes the various files.) That is, you copy the new index to
the server and then say "cat files > /dev/null". This pre-caches the
interesting files into memo
Hi Dan,
I think this may be your problem:
> Every day we produce a new dataset of 40 GB and have to switch one for the
> othe
If you really replace an index with a new index one a day, you throw away all
the hard work the OS has been doing to cache hot parts of your index in
memory. It takes
You say warming queries didn't help? How do those look like? Make sure you
facet and sort in all of the fields that your application allow
faceting/sorting. The same with the filters. Uninversion of fields is done
only when you commit, but warming queries should help you here.
Tomás
On Fri, Jan 27
Dan,
I can suggest a solution that should help. VeloBit enables you to add SSDs
to your servers as a cache (SSD will cost you $200, per server should be
enough). Then, assuming a 100MB/s read speed from your SAS disks, you can
read 50GB data into the VeloBit HyperCache cache in about 9 mins (this
