Re: need help from hard core solr experts - out of memory error

2014-04-20 Thread Mikhail Khludnev
I noticed enormous number of commits, which reasonably triggers merges that hits OOMe. Try to disable autocommits completely. Monitor commit occurrences in the log. On Sun, Apr 20, 2014 at 9:12 PM, Candygram For Mongo < candygram.for.mo...@gmail.com> wrote: > We have tried using fetchSize and we

Re: need help from hard core solr experts - out of memory error

2014-04-20 Thread Shawn Heisey
On 4/20/2014 11:12 AM, Candygram For Mongo wrote: > We have tried using fetchSize and we still got the same out of memory > errors. It needs to be batchSize, not fetchSize. I mentioned too much of the internal details. The fetchSize name is only important if you're writing source code that uses

Re: need help from hard core solr experts - out of memory error

2014-04-20 Thread Candygram For Mongo
We have tried using fetchSize and we still got the same out of memory errors. On Fri, Apr 18, 2014 at 9:39 PM, Shawn Heisey wrote: > On 4/18/2014 6:15 PM, Candygram For Mongo wrote: > > We are getting Out Of Memory errors when we try to execute a full import > > using the Data Import Handler.

Re: need help from hard core solr experts - out of memory error

2014-04-18 Thread Shawn Heisey
On 4/18/2014 6:15 PM, Candygram For Mongo wrote: > We are getting Out Of Memory errors when we try to execute a full import > using the Data Import Handler. This error originally occurred on a > production environment with a database containing 27 million records. Heap > memory was configured for

Re: need help from hard core solr experts - out of memory error

2014-04-18 Thread Candygram For Mongo
I have uploaded several files including the problem description with graphics to this link on Google drive: https://drive.google.com/folderview?id=0B7UpFqsS5lSjWEhxRE1NN2tMNTQ&usp=sharing I shared it with this address "solr-user@lucene.apache.org" so I am hoping it can be accessed by people in th

Re: need help from hard core solr experts - out of memory error

2014-04-18 Thread Candygram For Mongo
We consistently reproduce this problem on multiple systems configured with 6GB and 12GB of heap space. To quickly reproduce many cases for troubleshooting we reduced the heap space to 64, 128 and 512MB. With 6 or 12GB configured it takes hours to see the error. On Fri, Apr 18, 2014 at 5:54 PM,

Re: need help from hard core solr experts - out of memory error

2014-04-18 Thread Walter Underwood
I see heap size commands for 128 Meg and 512 Meg. That will certainly run out of memory. Why do you think you have 6G of heap with these settings? –Xmx128m –Xms128m –Xmx512m –Xms512m wunder On Apr 18, 2014, at 5:15 PM, Candygram For Mongo wrote: > I have lots of log files and other files to