Dominique, thanks, but I'm not sure the links you sent point to an actual solution. The Nginx logs, sometimes give a 499 return code which is: (499 Client Closed Request Used when the client has closed the request before the server could send a response.
but the timestamps of these log msgs do not coincide with the IOException, so I'm not sure they are related. Reinaldo On Mon, Aug 17, 2020 at 12:59 PM Dominique Bejean <dominique.bej...@eolya.fr> wrote: > Hi, > > It looks like this issues > https://github.com/eclipse/jetty.project/issues/4883 > https://github.com/eclipse/jetty.project/issues/2571 > > The Nginx server closed the connection. Any info in nginx log ? > > Dominique > > Le lun. 17 août 2020 à 17:33, Odysci <ody...@gmail.com> a écrit : > >> Hi, >> thanks for the reply. >> We're using solr 8.3.1, ZK 3.5.6 >> The stacktrace is below. >> The address on the first line "http://192.168.15.10:888/solr/mycollection"; >> is the "server" address in my nginx configuration, which points to 2 >> upstream solr nodes. There were no other solr or ZK messages in the logs. >> >> StackTrace: >> (Msg = IOException occured when talking to server at: >> http://192.168.15.10:888/solr/mycollection) >> org.apache.solr.client.solrj.SolrServerException: IOException occured >> when talking to server at: http://192.168.15.10:888/solr/mycollection >> at >> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:418) >> at >> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:754) >> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211) >> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1035) >> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1051) >> ... calls from our code >> ... calls from our code >> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >> at >> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) >> at java.base/java.lang.Thread.run(Thread.java:834) >> Caused by: java.nio.channels.AsynchronousCloseException >> at >> org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.close(HttpConnectionOverHTTP2.java:144) >> at >> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.onClose(HttpClientTransportOverHTTP2.java:170) >> at >> org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2$SessionListenerPromise.onClose(HttpClientTransportOverHTTP2.java:232) >> at org.eclipse.jetty.http2.api.Session$Listener.onClose(Session.java:206) >> at >> org.eclipse.jetty.http2.HTTP2Session.notifyClose(HTTP2Session.java:1153) >> at org.eclipse.jetty.http2.HTTP2Session.onGoAway(HTTP2Session.java:438) >> at >> org.eclipse.jetty.http2.parser.Parser$Listener$Wrapper.onGoAway(Parser.java:392) >> at >> org.eclipse.jetty.http2.parser.BodyParser.notifyGoAway(BodyParser.java:187) >> at >> org.eclipse.jetty.http2.parser.GoAwayBodyParser.onGoAway(GoAwayBodyParser.java:169) >> at >> org.eclipse.jetty.http2.parser.GoAwayBodyParser.parse(GoAwayBodyParser.java:108) >> at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:194) >> at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:123) >> at >> org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:248) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) >> at >> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170) >> at >> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125) >> at >> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348) >> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) >> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) >> at >> org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:68) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.invokeTask(EatWhatYouKill.java:345) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:300) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) >> at >> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) >> ... 3 more >> >> ----------------------------- >> >> I did consider using the solrJ cloud or lb clients, but nginx gives me >> more flexibility in controlling how the load balancing is done. I'm still >> running experiments to see which one works best for me. >> In the meantime, if you have any clues for why I'm getting this >> IOException, I'd appreciate. >> Thanks >> >> Reinaldo >> >> >> >> On Mon, Aug 17, 2020 at 10:59 AM Dominique Bejean < >> dominique.bej...@eolya.fr> wrote: >> >>> Hi, >>> >>> >>> >>> >>> >>> Can you provide more information ? >>> >>> >>> - Solr and ZK version >>> >>> >>> - full error stacktrace generated by SolrJ >>> >>> >>> - any concomitant and relevant information in solr nodes logs or zk logs >>> >>> >>> >>> >>> >>> Just to know, why not use a load balanced LBHttp... Solr Client ? >>> >>> >>> >>> >>> >>> Regards. >>> >>> >>> >>> >>> >>> Dominique >>> >>> >>> >>> >>> >>> >>> >>> >>> Le lun. 17 août 2020 à 00:41, Odysci <ody...@gmail.com> a écrit : >>> >>> >>> >>> >>> >>> > Hi, >>> >>> >>> > >>> >>> >>> > We have a solrcloud setup with 2 solr nodes and 3 ZK instances. Until >>> >>> >>> > recently I had my application server always call one of the solr nodes >>> (via >>> >>> >>> > solrJ), and it worked just fine. >>> >>> >>> > >>> >>> >>> > In order to improve reliability I put an Nginx reverse-proxy load >>> balancer >>> >>> >>> > between my application server and the solr nodes. The response time >>> >>> >>> > remained almost the same but we started getting the msg: >>> >>> >>> > >>> >>> >>> > IOException occured when talking to server http://myserver.... >>> >>> >>> > >>> >>> >>> > every minute or so (very randomly but consistently). Since our code >>> will >>> >>> >>> > just try again after a few milliseconds, the overall system continues >>> to >>> >>> >>> > work fine, despite the delay. I tried increasing all nginx related >>> >>> >>> > timeout's to no avail. >>> >>> >>> > I've searched for this msg a lot and most replies seem to be related to >>> >>> >>> > ssl. >>> >>> >>> > We are using http2solrclient but no ssl to solr. >>> >>> >>> > Can anyone shed any light on this? >>> >>> >>> > >>> >>> >>> > Thanks! >>> >>> >>> > Reinaldo >>> >>> >>> > >>> >>> >>> >> >>
