SolrCloud running away with resources
I've set up a simple 10 node, 5 shard SolrCloud 4.3. I'm pushing just a few thousand documents into it. What I'm doing is rather write intensive 100x...more writes than reads. I've noticed that there seems to be an unbounded use of resources. I'm seeing a steadily increasing number of network connections (monitored via: netstat | wc -l, which return over 5,500 and growing about 50 per minute) and over 2,200 open file descriptors (as shown on the Solr dashboard). This seems like there is something not configured correctly. At some point, rather soon I'm afraid, I'll run out of resources. -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-running-away-with-resources-tp4067154.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud running away with resources
Thanks! I've found the cause for this problem was the hardware load balancer (F5 LTM) was creating thousands of connections. So it turns out that it had nothing to do with SolrCloud. However, now I have another problem for which I'll create another post. -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-running-away-with-resources-tp4067154p4067426.html Sent from the Solr - User mailing list archive at Nabble.com.
Getting tons of EofException with jetty/SolrCloud
I have a 10 node, 4.3 SolrCloud with 5 shards. It's a heavy write (100x more writes than reads) environment. After less than 10k docs in 20 minutes or so I get tons of EofExceptions from what appears to be the synchronization traffic between the nodes. When I do queries during this exception storm I get inconsistent number of rows for a simple *:* query. Seems to suggest things are not synchronizing. I've changed a handful of RHEL sysctl parameters like net.core.somaxconn=4096, net.ipv4.tcp_max_syn_backlog=8192 with no luck. This sure seems like something basic I'm missing. -- View this message in context: http://lucene.472066.n3.nabble.com/Getting-tons-of-EofException-with-jetty-SolrCloud-tp4067427.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Getting tons of EofException with jetty/SolrCloud
15000 false 1000 I think these are close to the default values...not sure if I changed them. These mean a hard commit every 15 seconds...right? Seems sort of reasonable since we get a few hundred doc inserts in 15 seconds. Not sure...any advice is very welcome. -- View this message in context: http://lucene.472066.n3.nabble.com/Getting-tons-of-EofException-with-jetty-SolrCloud-tp4067427p4067433.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Getting tons of EofException with jetty/SolrCloud
Wonderful discussion, but it seems that the exact values here should only affect the freshness of the search results and the growth of the logs. What I have going on in a very simple, 10 node SolrCloud with quite low insert rates (10K docs/20 minutes) absolutely kills the cloud in the first 20 minutes. All the nodes spew out the EofException, searches provide inconsistent results, and the whole thing deteriorates into a bowl of mush. This is a simple and small use case...I really must have something wildly misconfigurated. -- View this message in context: http://lucene.472066.n3.nabble.com/Getting-tons-of-EofException-with-jetty-SolrCloud-tp4067427p4067457.html Sent from the Solr - User mailing list archive at Nabble.com.
