If I do _not_ have any auto-commit enabled, and add 500k documents and
commit at end, no problem.
If I instead set auto-commit maxDocs to 100000 (pretty large number),
and try to add 500k docs, with autocommits theoretically happening every
100k... I run into an OutOfMemory error.
Can anyone think of any reasons that would cause this, and how to
resolve it?
All I can think of is that in the first case, my newSearcher and
firstSearcher warming queries don't run until the 'document add' is
completely done. In the second case, there are newSearcher and
firstSearcher warming queries happening at the same time another process
is continuing to stream 'add's to Solr. Although at a maxDocs of
100000, I shouldn't (I think) get _overlapping_ warming queries, the
warming queries should be done before the next commit. I think. But
nonetheless, just the fact that warming queries are happening at the
same time 'add's are continuing to stream, could that be enough to
somehow increase memory usage enough to run into OOM?
- OutOfMemory and auto-commit Jonathan Rochkind
-
