On 5/9/2012 7:01 AM, richard.pog...@holidaylettings.co.uk wrote:
We are testing an updated version of our Solr server running solr 3.5.0 and we
are experiencing some performance issues with regard to updates and commits.
Searches are working well.
There are approximately 80,000 documents and the index is about 2.5 GB. This
does not seem to be extreme based on other implementations I have seen.
There seems to be no such thing as typical. My 11.5 million document
indexes are nearing 21 GB. Your documents are huge when compared to
mine, but I'm sure that someone out there has even larger documents.
An attempt to add a single document and commit is taking many minutes but the
time taken is not consistent.
Despite being in very light usage the java process seems to consistently
require an entire CPU core.
Optimising the index produced a limited improvement but this also takes a good
deal of time to run.
I am at a slight loss as to what the issue may be, I am interested in what
others think would be a normal commit time with this sort of data volume.
I am thinking there may be an issue with the schema but I am just guessing.
Frequent optimizes should be avoided. Optimizing is not inherently bad,
but it should happen during off hours, and index updates/deletes should
be suspended until the optimize is complete.
I believe it's fairly normal for a commit operation to use one core, at
least with 3.x. I have not yet used trunk, so I cannot say whether the
situation will be better on 4.0.
Without concrete information, I would guess that it's cache warming
that's making it go slow. You can see the total warmupTime on the
Statistics page in the admin GUI:
http://HOST:PORT/solr/CORENAME/admin/stats.jsp
Once there, search the page for warmup. The first entries found (in the
Core section) will be the total warmup time for the entire searcher.
Searching further, you will find additional entries in the Cache
section, those will tell you which of the caches is taking the majority
of the warmupTime.
If it is indeed the warmup that is going slowly, you can make it go
faster by reducing the autowarm counts for your caches. If you find (as
I have) that even a small number of warming queries still goes slowly,
you'll need to see if you can identify the slow queries before they get
executed and use LocalParams options to not cache those queries.
If the total warmupTime is low, then you can ignore everything I've said
above and keep looking for the cause, in which case answers to the
following questions may be helpful:
How often are you updating your index? How many documents are in a
typical update? Are you using explicit commits, autoCommit, or perhaps
both? If autoCommit is enabled, what are the settings?
Another thing that results in very slow queries is not having enough
free memory to cache the majority (or entirety) of the index.
Thanks,
Shawn
