Jasper, The simple answer is to increase -Xmx :) What is your ramBufferSizeMB (solrconfig.xml) set to? Default is 32 (MB).
That autocommit you mentioned is a DB commit? Not Solr one, right? If so, why is commit needed when you *read* data from DB? Otis ---- Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm ----- Original Message ----- > From: Jasper Floor <jasper.fl...@m4n.nl> > To: solr-user@lucene.apache.org > Cc: > Sent: Thursday, May 10, 2012 9:06 AM > Subject: Suddenly OOM > > Hi all, > > we've been running Solr 1.4 for about a year with no real problems. As > of monday it became impossible to do a full import on our master > because of an OOM. Now what I think is strange is that even after we > more than doubled the available memory there would still always be an > OOM. We seem to have reached a magic number of documents beyond which > Solr requires infinite memory (or at least more than 2.5x what it > previously needed which is the same as infinite unless we invest in > more resources). > > We have solved the immediate problem by changing autocommit=false, > holdability="CLOSE_CURSORS_AT_COMMIT", batchSize=10000. Now > holdability in this case I don't think does very much as I believe > this is the default behavior. BatchSize certainly has a direct effect > on performance (about 3x time difference between 1 and 10000). The > autocommit is a problem for us however. This leaves transactions > active in the db which may block other processes. > > We have about 5.1 million documents in the index which is about 2.2 gigabytes. > > A full index is a rare operation with us but when we need it we also > need it to work (thank you captain obvious). > > With the settings above a full index takes 15 minutes. We anticipate > we will be handling at least 10x the amount of data in the future. I > actually hope to have solr 4 by then but I can't sell a product which > isn't finalized yet here. > > > Thanks for any insight you can give. > > mvg, > Jasper >
