On 11/5/2013 10:16 AM, Luis Cappa Banda wrote:
I have a master-slave replication (Solr 4.1 version) with a 30 seconds
polling interval and continuously new documents are indexed, so after 30
seconds always new data must be replicated. My test index is not huge: just
5M documents.
I have experimented that a simple "q=*:*" query appears to be very slow (up
to 10 secs of QTime). After that first slow query the following "q=*:*"
queries are much quicker. I feel that warming up caches after replication
has something to say about this weird behavior, but maybe an index re-built
is also involved.
Question time:
*1.* How can I warm up caches against? There exists any solrconfig.xml
searcher to configure to be executed after replication events?
*2. *My system needs to execute queries to the slaves continuously. If
there exists any warm up way to reload caches, some queries will experience
slow response times until reload has finished, isn't it?
*3. *After a replication has done, does Solr execute any index rebuild
operation that slow down query responses, or this poor performance is just
due to caches?
*4. *My system is always querying by the latest documents indexed (I'm
filtering by document dates), and I don't use "fq" to execute that queries.
In this scenario, do you recommend to disable caches?
I suspect that you may be running into a situation where you don't have
enough OS disk cache for your index. When you replicate, the new data
that has just been replicated pushes existing data out of the cache.
You run your query that is slow, and the *Solr* caches (not the same
thing as the OS disk cache) get populated, making later queries fast.
You should be able to configure autowarming on your Solr caches to help
with this, but be aware that autowarming can be time-consuming, and if
you have replications happening potentially every 30 seconds, you may
find that your autowarming is taking more time than that. This can lead
to other problems.
If the amount of disk space taken up by those 5 million documents is
significantly larger than the amount of memory available on the server
that is not allocated directly to programs like Solr itself, then the
only true solution will be to add memory to the server.
Thanks,
Shawn
