On 11/1/2019 7:20 AM, Joe Obernberger wrote:
Hi All - getting this error from only one server in a 45 node cluster
when calling COLSTATUS. Any ideas?
<snip>
2019-11-01 13:17:32.556 INFO (qtp694316372-44709) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={name=UNCLASS_2019_1_18_36&action=COLSTATUS&wt=javabin&version=2}
status=0 QTime=94734
2019-11-01 13:17:32.567 INFO (qtp694316372-44688) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={name=UNCLASS_2021_2_17_36&action=COLSTATUS&wt=javabin&version=2}
status=0 QTime=815338
2019-11-01 13:17:32.570 INFO (qtp694316372-44688) [ ]
o.a.s.s.HttpSolrCall Unable to write response, client closed connection
or we are shutting down => org.eclipse.jetty.io.EofException: Closed
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491)
org.eclipse.jetty.io.EofException: Closed
Jetty's EofException almost always means one specific thing. The client
closed the connection before Solr could respond, so when Solr finally
finished processing and tried to have Jetty send the response, there was
nowhere to send it -- the connection was gone.
The first two lines of the log snippet indicate that there was one
COLSTATUS call that took nearly 95 seconds, and one that took 815
seconds, which is close to 15 minutes. Apparently those two calls
completed at the same time, even though they did not start at the same
time. Which suggests that for many minutes, the Solr server has been
under severe stress that prevented it from responding quickly to the
COLSTATUS request. The client gave up and closed its connection ...
probably on the one that took almost 15 minutes.
I found that the COLSTATUS action is mentioned in the 8.1 reference
guide on the "Collections API" page, but it is not there on the same
page in the 8.2 guide. That page in 8.2 appears to be significantly
smaller and missing most of the documentation that 8.1 has. So I think
we had a problem with documentation generation on 8.2.
Based on what I can see in the response on the 8.1 guide, I'm betting
that gathering COLSTATUS takes a fair amount of processing, and any
performance issues will make it very slow.
Thanks,
Shawn
