On 9/20/2012 1:02 AM, Alok Bhandari wrote:
Hello,
I am using solr 3.6.0 , I have observed many connection in CLOSE_WAIT state
after using solr server for some time. On further analysis and googling
found that I need to close the idle connections from the client which is
connecting to solr to query data and it does reduce the number of CLOSE_WAIT
connections but still some connection remain in that state.
I am using 2 shards and one observation is that if I don't use shards then I
am getting 0 CLOSE_WAIT connections. Need help of this as we need to use
distributed search using shards.
This is a "me too" email. We've got haproxy sitting in front of Solr.
A little more than half of the connections in the various WAIT states
are from the webserver SolrJ clients to haproxy listening on ports 8983
and 8984. This is SolrJ to haproxy, so there is a non-Solr component.
The other connections in WAIT states are connections to Solr listening
on port 8981. Because haproxy runs on one of the Solr servers, the
netstat table can't tell me which of those connections are from haproxy
and which are distributed search, but when I check the lsof output for
WAIT states on port 8981, the connections owned by haproxy are never in
a WAIT state. All of the connections in WAIT states are owned by java.
One of the java processes is Solr, the other is a SolrJ application that
handles index updates and deletes.
This is looking like a problem in SolrJ, Solr, or perhaps both. My
setup is not large enough for these connections to really be a problem,
but in a big enough installation, you might find the machine starved for
TCP ports.
netstat output:
http://dpaste.com/hold/804126/
lsof and ps output:
http://dpaste.com/hold/804127/
I've got four servers here, all running Solr -- idxa1, idxa2, idxb1, and
idxb2. The haproxy load balancer and the SolrJ indexing app are
currently running on idxa1, which is also where I obtained the netstat,
lsof, and ps output. The load balancer talks to a special broker core
which contains no index and has the shards parameter in the request
handler definition. The requests made by the SolrJ clients in the
webservers do not include a shards parameter.
[root@idxa1 solr]# java -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
[root@idxa1 solr]# uname -a
Linux idxa1 2.6.32-131.17.1.el6.x86_64 #1 SMP Thu Oct 6 19:24:09 BST
2011 x86_64 x86_64 x86_64 GNU/Linux
Thanks,
Shawn
