Thanks Shawn, I just regressed to Solrj 4.6.1 with http client 4.2.6 and am trying to reproduce the problem. Using YourKit to profile and even just manually simulating a few users at once, I see the same problem of open sockets. 6 sockets opened to the solr server and 2 of them still open after all is done and there is no server activity. Although this could be sockets kept in a connection pool.
On Thu, Feb 13, 2014 at 4:11 PM, Shawn Heisey <s...@elyograg.org> wrote: > On 2/13/2014 1:38 PM, Jared Rodriguez wrote: > >> I am using solr/solrj 4.6.1 along with the apache httpclient 4.3.2 as part >> of a web application which connects to the solr server via solrj >> using CloudSolrServer(); The web application is wired up with Guice, and >> there is a single instance of the CloudSolrServer class used by all >> inbound >> requests. All this is running on Amazon. >> >> Basically, everything looks and runs fine for a while, but even with >> moderate concurrency, solrj starts leaving sockets open. We are handling >> only about 250 connections to the web app per minute and each of these >> issues from 3 - 7 requests to solr. Over a 30 minute period of this type >> of use, we end up with many 1000s of lingering sockets. I can see these >> when running netstats >> >> tcp 0 0 ip-10-80-14-26.ec2.in:41098ip-10-99-145-47.ec2.i:glrpc >> TIME_WAIT >> >> All to the same target host, which is my solr server. There are no other >> pieces of infrastructure on that box, just solr. Eventually, the server >> just dies as no further sockets can be opened and the opened ones are not >> reused. >> >> The solr server itself is unphased and running like a champ. Average >> timer >> per request of 0.126, as seen in the solr web app admin UI query handler >> stats. >> >> Apache httpclient had a bunch of leakage from version 4.2.x that they >> cleaned up and refactored in 4.3.x, which is why I upgraded. Currently, >> solrj makes use of the old leaky 4.2 classes for establishing connections >> and using a connection pool. >> > > This is something that I can look into. > > I have a SolrJ program with SolrJ 4.5.1 and HttpClient 4.3.1 that does not > leak anything. I thought it was migrated already to SolrJ 4.6.1, but now > that I know it's not, I will upgrade SolrJ first and then HttpClient, and > see whether I have the same problem with either upgrade. > > I am using HttpSolrServer, not CloudSolrServer, because the Solr servers > are not running SolrCloud. CloudSolrServer ultimately uses HttpSolrServer > for its communication, so my initial thought is that this is not important, > but we'll see. > > In version 4.7, Solr will include HttpClient 4.3.1. See SOLR-5590. > > https://issues.apache.org/jira/browse/SOLR-5590 > > A question for committers with a lot of experience: Do we have any tests > that check for connection leaks? > > Thanks, > Shawn > > -- Jared Rodriguez
