Kiran & Shawn, Thank you both for the info and you are both absolutely correct. The issue was not that sockets were leaked, but that wait time thing is a killer. I ended up fixing the problem by changing the system property of "http.maxConnections" which is used internally to Apache httpclient to setup the PoolingClientConnectionManager. Previously, this had no value, and was defaulting to 5. That meant that any time there were more than 50 (maxConnections * maxperroute) concurrent connections to the Solr server, non reusable connections were opening and closing and thus sitting in that idle state .. too many sockets.
The fix was simply tuning the pool and setting "http.maxConnections" to a higher value representing the number of concurrent users that I expect. Problem fixed, and a modest speed improvement simply by higher socket reuse. Thank you both for the help! Jared On Mon, Feb 17, 2014 at 3:03 AM, Kiran Chitturi < kiran.chitt...@lucidworks.com> wrote: > Jared, > > I faced a similar issue when using CloudSolrServer with Solr. As Shawn > pointed out the 'TIME_WAIT' status happens when the connection is closed > by the http client. HTTP client closes connection whenever it thinks the > connection is stale > ( > https://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html > #d5e405). Even the docs point out the stale connection checking cannot be > all reliable. > > I see two ways to get around this: > > 1. Enable 'SO_REUSEADDR' > 2. Disable stale connection checks. > > Also by default, when we create CSS it does not explicitly configure any > http client parameters > ( > https://github.com/apache/lucene-solr/blob/trunk/solr/solrj/src/java/org/a > pache/solr/client/solrj/impl/CloudSolrServer.java#L124). In this case, the > default configuration parameters (max connections, max connections per > host) are used for a http connection. You can explicitly configure these > params when creating CSS using HttpClientUtil: > > ModifiableSolrParams params = new ModifiableSolrParams(); > params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 128); > params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 32); > params.set(HttpClientUtil.PROP_FOLLOW_REDIRECTS, false); > params.set(HttpClientUtil.PROP_CONNECTION_TIMEOUT, 30000); > httpClient = HttpClientUtil.createClient(params); > > final HttpClient client = HttpClientUtil.createClient(params); > LBHttpSolrServer lb = new LBHttpSolrServer(client); > CloudSolrServer server = new CloudSolrServer(zkConnect, lb); > > > Currently, I am using http client 4.3.2 and building the client when > creating the CSS. I also use 'SO_REUSEADDR' option and I haven't seen the > 'TIME_WAIT' after this (may be because of better handling of stale > connections in 4.3.2 or because of 'SO_REUSEADDR' param enabled). My > current http client code looks like this: (works only with http client > 4.3.2) > > HttpClientBuilder httpBuilder = HttpClientBuilder.create(); > > Builder socketConfig = SocketConfig.custom(); > socketConfig.setSoReuseAddress(true); > socketConfig.setSoTimeout(10000); > httpBuilder.setDefaultSocketConfig(socketConfig.build()); > httpBuilder.setMaxConnTotal(300); > httpBuilder.setMaxConnPerRoute(100); > > httpBuilder.disableRedirectHandling(); > httpBuilder.useSystemProperties(); > LBHttpSolrServer lb = new LBHttpSolrServer(httpClient, parser) > CloudSolrServer server = new CloudSolrServer(zkConnect, lb); > > > There should be a way to configure socket reuse with 4.2.3 too. You can > try different configurations. I am surprised you have 'TIME_WAIT' > connections even after 30 minutes because 'TIME_WAIT' connection should be > closed by default in 2 mins by O.S I think. > > > HTH, > > -- > Kiran Chitturi, > > > On 2/13/14 12:38 PM, "Jared Rodriguez" <jrodrig...@kitedesk.com> wrote: > > >I am using solr/solrj 4.6.1 along with the apache httpclient 4.3.2 as part > >of a web application which connects to the solr server via solrj > >using CloudSolrServer(); The web application is wired up with Guice, and > >there is a single instance of the CloudSolrServer class used by all > >inbound > >requests. All this is running on Amazon. > > > >Basically, everything looks and runs fine for a while, but even with > >moderate concurrency, solrj starts leaving sockets open. We are handling > >only about 250 connections to the web app per minute and each of these > >issues from 3 - 7 requests to solr. Over a 30 minute period of this type > >of use, we end up with many 1000s of lingering sockets. I can see these > >when running netstats > > > >tcp 0 0 ip-10-80-14-26.ec2.in:41098 > >ip-10-99-145-47.ec2.i:glrpc > >TIME_WAIT > > > >All to the same target host, which is my solr server. There are no other > >pieces of infrastructure on that box, just solr. Eventually, the server > >just dies as no further sockets can be opened and the opened ones are not > >reused. > > > >The solr server itself is unphased and running like a champ. Average > >timer > >per request of 0.126, as seen in the solr web app admin UI query handler > >stats. > > > >Apache httpclient had a bunch of leakage from version 4.2.x that they > >cleaned up and refactored in 4.3.x, which is why I upgraded. Currently, > >solrj makes use of the old leaky 4.2 classes for establishing connections > >and using a connection pool. > > > > > http://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-4.3.x.t > >xt > > > > > > > >-- > >Jared Rodriguez > > -- Jared Rodriguez
