On 1/31/2013 1:01 AM, Mou wrote:
I am running solr 3.4 on tomcat 7.
Our index is very big , two cores each 120G. We are searching the slaves
which are replicated every 30 min.
I am using filtercache only and We have more than 90% cache hits. We use
lot of filter queries, queries are usually pretty big with 10-20 fq
parameters. Not all filters are cached.
we are searching three shards and query looks like this --
shards=core1,core2,core3&q=*:* &fq=field1:some value&fq = -field2=some
value&sort=date
But some queries are taking more than 30 sec to return result and the
behavior is intermittent. I can not find relation to replication. We are
using Zing jvm which reduced our GC pause to milli secs, so GC is not a
problem.
Complex queries, especially on a distributed search, can be very slow.
In my experience, uncached filters make things particularly slow.
Your first paragraph says you have two cores 120GB each, but then later
you say you are using three cores in a shards parameter. What's the
true core situation?
If you have a total index size for this JVM of 240GB, then you may not
have enough RAM to let the OS disk cache work efficiently. For that
size of index, I would plan on a system with at least 128GB of RAM,
256GB would be better. You have to have enough free memory (after the
OS and programs including tomcat/solr) in the system to cache the
critical pieces of your index.
Tomcat would probably need a heap size between 8GB and 24GB - it's
impossible to give you the right heap size here, you'd just have to
test. I could be way off on that estimate, too.
To give you an idea of how to size memory based on a production Solr
3.5.0 system with good performance, one of my solr servers has 70GB of
total index data, of which 24GB is stored fields and 22GB is
termvectors. The OS disk cache has 41.8GB of data in it at the moment.
The system has 64GB of memory and Solr (Jetty) has a max heap size of
8GB. Based on observations, I could run with a heap size of 4GB during
normal operation, but when I am doing a full database import on my
indexes, it requires the 8GB heap.
I looked through the mailing list history to see what else you've said
about your setup and what other help you've gotten. In one of your
other messages, you said that you have a 70GB heap size. That is
extremely large, and probably not necessary. If you have found that it
is necessary, then your overall Solr architecture may need further
adjustment.
One of your earlier messages indicated that you are using SSD storage.
Main system memory is quite a lot faster than an SSD, so the OS cache is
still important.
Thanks,
Shawn
