I'm not a JVM nor kernel expert so take this with a pinch of salt... my guess woule be: Having more CLOSE_WAIT connections can *probably* affect you in two ways:
1. Longer GC Cycles pauses (not sure how much of the connection the JVM handles) 2. Some of the connections might be put on the tcp-backlog Here is a good article about tcp-tunning http://engineering.chartbeat.com/2014/01/02/part-1-lessons-learned-tuning-tcp-and-nginx-in-ec2/ On 23 February 2016 at 13:54, Niraj Aswani <niraj.asw...@klevu.com> wrote: > Hi Jacques, > > Thank you for your reply. I'll give these parameters a try and report back. > Do you think having higher number of CLOSE_WAIT connections affects the > over all search delivery speed? > > Regards, > Niraj > > On Tue, Feb 23, 2016 at 2:57 PM, Jacques du Rand <jacq...@pricecheck.co.za > > > wrote: > > > Try to fiddle with your tcp settings: > > > > /etc/sysctl.conf > > net.ipv4.tcp_fin_timeout = 30 > > net.ipv4.tcp_keepalive_intvl = 2 > > net.ipv4.tcp_keepalive_probes = 2 > > net.ipv4.tcp_keepalive_time = 180 > > > > That should help. > > > > > > On 23 February 2016 at 07:13, Niraj Aswani <nirajasw...@gmail.com> > wrote: > > > > > Hi, > > > > > > I am on solr 4.8.1 and running master-slave setup with lots of cores > > (>3K). > > > Internally I maintain an instance of HTTPSolrServer for each core that > is > > > reused for querying the respective cores. A request is received by an > > > intermediary tomcat and forwarded to another tomcat running Solr. > > > > > > Over the period we see high search latency. Some requests start to take > > too > > > long and eventually result into timeouts. > > > > > > Investigating this, I see that, over the period, a high number of > > > CLOSE_WAIT sockets (>3300) are building up. Running `netstat -p` seems > to > > > suggest that these sockets were initiated by the intermediary tomcats > > when > > > communicating to the Solr. > > > > > > Questions are: > > > > > > - Why do we see such high number of CLOSE_WAiT sockets? Shouldn't the > > > HTTPSolrServer take care of closing these connections after > communicating > > > with the Solr server? > > > > > > - Does the high number of CLOSE_WAIT have anything to do with search > > > latency? > > > > > > Any suggestion on the matter is highly appreciated! > > > > > > Regards, > > > Niraj > > > > > > > > > > > -- > > Jacques du Rand > > Senior R&D Programmer > > > > T: +27214688017 > > F: +27862160617 > > E: jacq...@pricecheck.co.za > > < > > > https://mail.naspers.com/owa/redir.aspx?SURL=uQ0aY47BwKkgYjLPs4CuWJgrQlRUZlf73Ere0TGrauf5NrymvPnSCG0AYQBpAGwAdABvADoAagBhAGMAcQB1AGUAcwBAAHAAcgBpAGMAZQBjAGgAZQBjAGsALgBjAG8ALgB6AGEA&URL=mailto%3ajacques%40pricecheck.co.za > > > > > > -- Jacques du Rand Senior R&D Programmer T: +27214688017 F: +27862160617 E: jacq...@pricecheck.co.za <https://mail.naspers.com/owa/redir.aspx?SURL=uQ0aY47BwKkgYjLPs4CuWJgrQlRUZlf73Ere0TGrauf5NrymvPnSCG0AYQBpAGwAdABvADoAagBhAGMAcQB1AGUAcwBAAHAAcgBpAGMAZQBjAGgAZQBjAGsALgBjAG8ALgB6AGEA&URL=mailto%3ajacques%40pricecheck.co.za>
