Hi Shawn, Thanks for the helpful and thorough response. While I understand all of the factors that you've outlined for memory requirements (in fact, I'd previously read your page on Solr performance problems), it is baffling to me why two identical SolrCloud instances, each sharded across 3 machines with identical hardware, would run into these memory issues at such differently memory limits (one SolrCloud instance started seeing OOM issues at 2 million indexed documents, the other started seeing OOM issues between 20 and 30 million indexed documents).
When I stated that approximately 1.5GB, I mean that this is how much heap space I allocated when launching java with -Xmx, and I can see the java process using that full amount of RAM. >From a usage perspective, the usage doesn't seem all that heavy. I'm indexing about 600k documents an hour (each of which have ~20 short numeric or string fields). I have the autoSoftCommit parameter set for once a second, and the autoCommit time set for every 5 minutes, with openSearcher set to false. Finally, I have maxWarmingSearchers at 2. Besides indexing those documents, I've been doing a few small queries just to check how many documents have been indexed, and a few other small queries, sorting by a single attribute. These searches are very infrequent though, maybe 5 or 6 an hour. Seems like a strange issue indeed. My expectation is that Solr would hit a point where it becomes horribly slow after some threshold where things don't fit in the cache, but I'd never expect it to simply crash like it's doing. -- View this message in context: http://lucene.472066.n3.nabble.com/SolrException-Error-when-indexing-new-documents-at-scale-in-SolrCloud-tp4111551p4111680.html Sent from the Solr - User mailing list archive at Nabble.com.
