: > How big is your physical index directory on disk? : It's about 2.9G now. : Is there a direct connection between size of index and usage of ram?
Generally yes. Lucene loads a lot of your index into memory ... not neccessarily the "stored" fields, but quite a bit of the index structure needed forsearching ... not to mention things like the FieldCache if you start sorting by specific fields. You may want to consult the lucene user list to get more advice about the minimum amount of RAM recommended just for using a lucene index of size ___G in a JVM ... personally, I've always made sure my JVM was allocated a max Heap size at least twice as big as my acctual index ... mainly a paranoia thing on my part. : Hm, after disabling all caches I still get OutOfMemoryErrors. : All I do currently while testing is to delete documents. No searching or : inserting. Typically after deleting about 20,000 documents the server : throws the first error message. interesting .. are you getting the OutOfMemory on an actual delete operation or when doing a commit after executing some deletes? part of the problem may be that under the covers, any delete involves doing a query (even if oyou are deleting by uniqueKey, that's implimented as a delete by Term, which requires iterating over a TermEnum to find the relevent document, and if your index is big enough, loading that TermEnum and may be the cause of your OOM. : One thing that appears suspicious to me is that everything went fine as : long as the number of documents was below 10 million. Problems started : when this limit was exceeded. But maybe this is just a coincidence. Maybe, maybe not ... what options are you using in your solrconfig.xml's indexDefaults and mainIndex blocks? ... 10 million documents could be the magic point at which your mergeFactor triggers the merging of several large segments into one uber segment -- which may be big enough to cause an OOM when the IndexReader tries to open it. (Doug, Yonik, and Erik understand the underlying lucene memory usange better then i do, hopefully they'll chime in with some advice) -Hoss
