On 4/23/2013 11:27 AM, gustavonasu wrote:
We migrated recently from Solr 1.4 to 3.6.1. In the new version we have
noticed that after some hours (around 8) the autocommit is taking more time
to be executed.
In the new version we have noticed that after some hours the autocommit
is taking more time to be executed. We configured autocommit with maxDocs=50
and maxTime=10000ms but we've gotten few (3-5) minutes to index documents (I
got this time seeing the docsPending on the Update Stats and refresh page.
Is there another way to verify that information?).
Your question is a bit jumbled so I don't know exactly what you are
saying for all of this, but I'll attempt to answer what I can. Usually
if your commits are taking a really long time, it means you're running
into one of two problems:
1) It is taking a really long time to autowarm your Solr caches. In
most cases, it is the filterCache that takes the time, but not always.
You can see how long it takes to warm the entire searcher as well as
each individual cache in the Statistics page of the admin UI. To fix
this, you have to reduce the autowarmCount on your caches, reduce the
complexity of your queries and filters or both.
2) Your Java heap is getting exhausted and Java is spending too much
time doing full garbage collections so it can keep working. Eventually
this problem will result in OOM (Out of Memory) errors in your Solr log.
To fix this, raise your max heap, which is the -Xmx java option when
starting your servlet container. Raising the java heap might also
require that you add physical RAM to your server.
On version 3.6, I believe that an index update/commit that results in
segment merging will wait for that merging to complete. If you do a lot
of indexing, eventually you will run into a very large merge, and that
can take a lot of time. This would not explain why every autoCommit is
taking a long time, though - it would only explain one out of dozens or
hundreds.
A similar problem has been happening with the replication. We configured
the pollInterval with 60s but the replication takes some minutes to be
executed. You could see the timeElapsed value (around 6 minutes) on the
Replication Stats.
If you optimize your index, or do enough index updates so that a large
merge takes place, then a very large portion of your index will be
comprised of brand new files, and if your index is large, that can take
a long time to replicate. It is also possible for the java heap problem
(mentioned above) to cause this.
Thanks,
Shawn
