Chris Hostetter wrote: > interesting .. are you getting the OutOfMemory on an actual delete > operation or when doing a commit after executing some deletes?
Yes, on a delete operation. I'm not doing any commits until the end of all delete operations. After reading this I was curious if using commits during deleting would have any effect. So I tested doing a commit after 10,000 deletes at a time (which, I know, is not recommended). But that simply didn't change anything. Meanwhile I found out that I can gain 10,000 documents more to delete (before getting an OOM) by increasing the heap space by 500M. Unfortunately we need to delete about 200,000 documents on each update which would need 10G to be added to the heap space. Not to speak of the same number of inserts. > part of the problem may be that under the covers, any delete involves > doing a query (even if oyou are deleting by uniqueKey, that's implimented > s a delete by Term, which requires iterating over a TermEnum to find the > relevent document, and if your index is big enough, loading that TermEnum > and may be the cause of your OOM. Yes, I thought so, too. And in fact I get OOM even if I just submit search queries. > Maybe, maybe not ... what options are you using in your solrconfig.xml's > indexDefaults and mainIndex blocks? I adopted the default values from the example installation which looked quite reasonable to me. > ... 10 million documents could be the > magic point at which your mergeFactor triggers the merging of several > large segments into one uber segment -- which may be big enough to cause > an OOM when the IndexReader tries to open it. Yes, I'm using the default mergeFactor of 10 and as 10 million is 10^7 this is what appeared suspicious to me. Is it right, that the mergeFactor connot be changed once the index has been built? Marcus
