Then most likely its due to closing of connection as mentioned above though you said it's not happening in that part of your code. To rule out firewall possibility, you can test in some other/local env. Also how many requests/client/connections happening concurrently.
Thanks, Susheel On Tue, Jul 18, 2017 at 8:43 AM, Markus Jelsma <markus.jel...@openindex.io> wrote: > Hello Susheel, > > No, nothing at all. I've check all six nodes, they are clean. > > Thanks, > Markus > > > > -----Original message----- > > From:Susheel Kumar <susheel2...@gmail.com> > > Sent: Tuesday 18th July 2017 14:30 > > To: solr-user@lucene.apache.org > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace > > > > Do you see any errors etc. in solr.log during this time? > > > > On Tue, Jul 18, 2017 at 7:10 AM, Markus Jelsma < > markus.jel...@openindex.io> > > wrote: > > > > > The problem was never resolved but Shawn asked for the stack trace, > here > > > it is: > > > > > > org.apache.solr.client.solrj.SolrServerException: java.lang. > IllegalStateException: > > > Connection pool shut down > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > doRequest(LBHttpSolrClient.java:485) > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request( > > > LBHttpSolrClient.java:388) > > > at org.apache.solr.client.solrj.impl.CloudSolrClient. > > > sendRequest(CloudSolrClient.java:1383) > > > at org.apache.solr.client.solrj.impl.CloudSolrClient. > > > requestWithRetryOnStaleState(CloudSolrClient.java:1134) > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request( > > > CloudSolrClient.java:1073) > > > at org.apache.solr.client.solrj.SolrRequest.process( > SolrRequest.java:160) > > > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942) > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > java:1173) > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > java:1090) > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > java:1110) > > > ..internal method calling getById().. > > > at java.lang.Thread.run(Thread.java:748) > > > Caused by: java.lang.IllegalStateException: Connection pool shut down > > > at org.apache.http.util.Asserts.check(Asserts.java:34) > > > at org.apache.http.pool.AbstractConnPool.lease( > AbstractConnPool.java:184) > > > at org.apache.http.pool.AbstractConnPool.lease( > AbstractConnPool.java:217) > > > at org.apache.http.impl.conn.PoolingClientConnectionManager > > > .requestConnection(PoolingClientConnectionManager.java:184) > > > at org.apache.http.impl.client.DefaultRequestDirector.execute( > > > DefaultRequestDirector.java:415) > > > at org.apache.http.impl.client.AbstractHttpClient.doExecute( > > > AbstractHttpClient.java:882) > > > at org.apache.http.impl.client.CloseableHttpClient.execute( > > > CloseableHttpClient.java:82) > > > at org.apache.http.impl.client.CloseableHttpClient.execute( > > > CloseableHttpClient.java:55) > > > at org.apache.solr.client.solrj.impl.HttpSolrClient. > > > executeMethod(HttpSolrClient.java:515) > > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > HttpSolrClient.java:279) > > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > HttpSolrClient.java:268) > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > doRequest(LBHttpSolrClient.java:447) > > > ... 24 more > > > > > > So, to summarize, we have a program checking presence of documents in > Solr > > > using getById() and we don't want this exception to bubble up, we want > > > SolrJ to restore the connection pool just as CloudSolrClient would > move on > > > to another node if one went down in the mean time. > > > > > > Is this possible? How? > > > > > > Many thanks, > > > Markus > > > > > > -----Original message----- > > > > From:Markus Jelsma <markus.jel...@openindex.io> > > > > Sent: Thursday 29th June 2017 16:38 > > > > To: solr-user@lucene.apache.org > > > > Subject: RE: SolrJ 6.6.0 Connection pool shutdown > > > > > > > > Thanks. I probably should have mentioned there is no firewall > limiting > > > connections between those hosts. Actually, the processes run on the > same > > > hosts as the Solr cluster is running on. > > > > > > > > Thanks, > > > > Markus > > > > > > > > > > > > > > > > -----Original message----- > > > > > From:Alexandre Rafalovitch <arafa...@gmail.com> > > > > > Sent: Thursday 29th June 2017 15:38 > > > > > To: solr-user <solr-user@lucene.apache.org> > > > > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > > > > > > > > > One thing to check is whether there is a firewall between the > client > > > > > and the server. They - sometimes - cut the silent connections in > the > > > > > _middle_ (at the firewall). The usual solution is keepAlive > request of > > > > > some kind or not using the connection pool. > > > > > > > > > > One way to check is with network tracer like Wireshark and checking > > > > > whether the actual hardware at the other end of the connection is a > > > > > normal server or some sort of unexpected hardware piece of > equipment > > > > > (firewall). Yes, that's using the hammer to swat a fly :-) > > > > > > > > > > Regards, > > > > > Alex. > > > > > ---- > > > > > http://www.solr-start.com/ - Resources for Solr users, new and > > > experienced > > > > > > > > > > > > > > > On 29 June 2017 at 08:21, Markus Jelsma < > markus.jel...@openindex.io> > > > wrote: > > > > > > Hi, > > > > > > > > > > > > Everything is 6.6.0. I could include a stack trace (i don't print > > > them in my program), but that would only be the the trace from > getById() to > > > CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that > > > what you're looking for? > > > > > > > > > > > > We haven't called close() in that particular part of the program. > > > > > > > > > > > > Method requestWithRetryOnStaleState has some retry logic built-in > > > but doesn't seem to work for the exception i got. > > > > > > > > > > > > I'll let it print the stack trace and get back if it happens > again. > > > > > > > > > > > > Thanks, > > > > > > Markus > > > > > > > > > > > > -----Original message----- > > > > > >> From:Shawn Heisey <apa...@elyograg.org> > > > > > >> Sent: Tuesday 27th June 2017 23:02 > > > > > >> To: solr-user@lucene.apache.org > > > > > >> Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > > > > >> > > > > > >> On 6/27/2017 6:50 AM, Markus Jelsma wrote: > > > > > >> > We have a proces checking presence of many documents in a > > > collection, just a simple client.getById(id). It sometimes begins > throwing > > > lots of these exceptions in a row: > > > > > >> > > > > > > >> > org.apache.solr.client.solrj.SolrServerException: java.lang. > IllegalStateException: > > > Connection pool shut down > > > > > >> > > > > > > >> > Then, as suddenly as it appeared, it's gone again a no longer > a > > > problem. I would expect SolrJ not to throw this but to wait until it > the > > > connection pool, or whatever mechanism is there, to recover. > > > > > >> > > > > > > >> > Did i miss a magic parameter for SolrJ?\ > > > > > >> > > > > > >> That error message will be much longer than what you've provided > > > here. > > > > > >> It will have a java stacktrace that's typically a dozen or so > lines > > > > > >> long. There may also be one or more "Caused by" sections after > the > > > > > >> stacktrace, each with a stacktrace of its own. Can you share > the > > > full > > > > > >> error message? Is the server also running 6.6.0, or a different > > > version? > > > > > >> > > > > > >> It would also be helpful if you can share the SolrJ code you've > > > written, > > > > > >> cleanly redacted to remove anything sensitive. > > > > > >> > > > > > >> That particular message ("Connection pool shut down") sounds > like it > > > > > >> probably came from HttpClient, which SolrJ uses ... and I would > > > expect > > > > > >> that to only happen if you close/shutdown the HttpClient or the > > > > > >> SolrClient. After closing, a client can't be used any more. > > > Normally > > > > > >> the only time you should close a client is right before exiting > the > > > > > >> program, although if the program's about to exit, it's generally > > > > > >> unnecessary, so in my opinion for *most* usages, closing the > client > > > > > >> likely never needs to happen. > > > > > >> > > > > > >> Thanks, > > > > > >> Shawn > > > > > >> > > > > > >> > > > > > > > > > > > > > > >
