Hi all,
We are using Solr over Jetty with a large index, sharded and distributed over
multiple machines. Our queries are quite long, involving boolean and proximity
operators. We cut the connection at the client side after 5 minutes. Also, we
are using parameter timeAllowed to stop executing it on the server after a
while.
We quite often run into situations when solr "blocks". The load on the server
increases and a thread dump on the solr process shows many threads like below:
"btpool0-49" prio=10 tid=0x00007f73afe1d000 nid=0x3581 runnable
[0x00000000451a0000]
java.lang.Thread.State: RUNNABLE
at java.io.PrintWriter.write(PrintWriter.java:362)
at org.apache.solr.common.util.XML.escape(XML.java:206)
at org.apache.solr.common.util.XML.escapeCharData(XML.java:79)
at org.apache.solr.request.XMLWriter.writePrim(XMLWriter.java:832)
at org.apache.solr.request.XMLWriter.writeStr(XMLWriter.java:684)
at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:564)
at org.apache.solr.request.XMLWriter.writeDoc(XMLWriter.java:435)
at org.apache.solr.request.XMLWriter$2.writeDocs(XMLWriter.java:514)
at org.apache.solr.request.XMLWriter.writeDocuments(XMLWriter.java:485)
at
org.apache.solr.request.XMLWriter.writeSolrDocumentList(XMLWriter.java:494)
at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:588)
at org.apache.solr.request.XMLWriter.writeResponse(XMLWriter.java:130)
at
org.apache.solr.request.XMLResponseWriter.write(XMLResponseWriter.java:34)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:325)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
......................................
A netstat on the machine shows sockets in state CLOSE_WAIT. However, they are
fewer than the number of RUNNABLE threads as the above.
Why is this happening? Is there anything we can do to avoid getting in these
situations?
Thanks,
roxana
