From: Lance Norskog <goks...@gmail.com>
Subject: Re: OutOfMemory and auto-commit
To: solr-user@lucene.apache.org
Date: Thursday, October 21, 2010, 9:53 PM
Yes. Indexing activity suspends until
the commit finishes, then
starts. Having both queries and indexing on the same Solr
will have
this memory problem.
Lance
On Thu, Oct 21, 2010 at 1:16 PM, Jonathan Rochkind <rochk...@jhu.edu>
wrote:
If I do _not_ have any auto-commit enabled, and add
500k documents and
commit at end, no problem.
If I instead set auto-commit maxDocs to 100000 (pretty
large number), and
try to add 500k docs, with autocommits theoretically
happening every 100k...
I run into an OutOfMemory error.
Can anyone think of any reasons that would cause this,
and how to resolve
it?
All I can think of is that in the first case, my
newSearcher and
firstSearcher warming queries don't run until the
'document add' is
completely done. In the second case, there are
newSearcher and firstSearcher
warming queries happening at the same time another
process is continuing to
stream 'add's to Solr. Although at a maxDocs of
100000, I shouldn't (I
think) get _overlapping_ warming queries, the warming
queries should be done
before the next commit. I think. But nonetheless, just
the fact that warming
queries are happening at the same time 'add's are
continuing to stream,
could that be enough to somehow increase memory usage
enough to run into
OOM?
--
Lance Norskog
goks...@gmail.com
