On 3/16/2014 10:34 AM, adfel70 wrote: > I have a 12-node solr 4.6.1 cluster. each node has 2 solr procceses, running > on 8gb heap jvms. each node has total of 64gb memory. > My current collection (7 shards, 3 replicas) has around 500 million docs. > I'm performing bulk indexing into the collection. I set softCommit to 10 > minutes and hardCommit openSearcher=false to 15 minutes.
How much index data does each server have on it? This would be the sum total of the index directories of all your cores. > I recently started seeing the following problems while indexing - every 10 > minutes ( and I assume that this is the 10minutes soft-commit cycles) I get > the following errors: > 1. EofExcpetion from jetty in HttpOutput.write send from SolrDispatchFilter > 2. queries to all cores start getting high latencies (more the 10 seconds) EofException errors happen when your client disconnects before the request is complete. I would strongly recommend that you *NOT* configure hard timeouts for your client connections, or that you make them really long, five minutes or so. For SolrJ, this is the SO_TIMEOUT. These problems sound like one of two things. It could be either or both: 1) You don't have enough RAM to cache your index effectively. With 64GB of RAM and 16GB heap, you have approximately 48GB of RAM left over for other software and the OS disk cache. If the total index size on each machine is in the neighborhood of 60GB (or larger), this might be a problem. If you have software other than Solr running on the machine, you must subtract it's direct and indirect memory requirements from the available OS disk cache. 2) Indexing results in a LOT of object creation, most of which exist for a relatively short time. This can result in severe problems with garbage collection pauses. Both problems listed above (and a few others) are discussed at the wiki page linked below. As you will read, there are two major causes of GC symptoms - a heap that's too small and incorrect (or nonexistent) GC tuning. With a very large index like yours, either or both of these GC symptoms could be happening. http://wiki.apache.org/solr/SolrPerformanceProblemshttp://wiki.apache.org/solr/SolrPerformanceProblems Side note: You should only be running one Solr process per machine. Running multiple processes creates additional memory overhead. Any hard limits that you might have run into with a single Solr process can be overcome with configuration options for Jetty, Solr, or the operating system. Thanks, Shawn
