If this has been working fine for a while and suddenly started this
behavior my first suspicion would be excessive GC, i.e. you've been adding
docs and your heap is no longer adequate. If Java needs to do a
stop-the-world garbage collection you can get these kinds of errors.
So I'd enable GC logging on my Solr instances (and ZooKeeper) and then use
something like GCViewer to figure out whether I have excessive
stop-the-world pauses first.
Best,
Erick
On Fri, Jan 5, 2018 at 11:36 AM, Satyaprashant Bezwada <
satyaprashant.bezw...@nasdaq.com> wrote:
> Not sure where we are going wrong in our implementation.
>
> We have a Solr cloud environment (Solr 6.5), with 2 solr nodes and 3
> Zookeeper servers. The environment was running without any issues, but
> lately noticed that one of the Solr node keeps shutting down frequently. We
> have replication in place where in we stated one per node.
>
> This is what I see in the logs and it happens only on one particular Solr
> node, which keeps shutting down frequently.
>
> 2018-01-04 07:34:59.168 INFO (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.c.PluginBag Going to create a
> new requestHandler with {type = requestHandler,name = /update/extract,class
> = solr.extraction.ExtractingRequestHandler,attributes = {startup=lazy,
> name=/update/extract, class=solr.extraction.ExtractingRequestHandler},args
> = {defaults={uprefix=ignored_,captureAttr=true,df=_text_}}}
> 2018-01-04 07:35:00.490 ERROR (updateExecutor-2-thread-9-
> processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2
> x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB)
> [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1]
> o.a.s.u.StreamingSolrClients error
> java.net.ConnectException: Connection refused
>
>
> 2018-01-04 07:35:00.493 ERROR (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor
> forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed
> - retrying ... retries: 1 add{,id=0DODD:S8IYAB-
> 13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=
> TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/
> rsp:-1:java.net.ConnectException: Connection refused
>
> 2018-01-04 07:35:00.995 ERROR (updateExecutor-2-thread-9-
> processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2
> x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB)
> [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1]
> o.a.s.u.StreamingSolrClients error
> java.net.ConnectException: Connection refused
>
> 2018-01-04 07:35:00.996 ERROR (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor
> forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed
> - retrying ... retries: 2 add{,id=0DODD:S8IYAB-
> 13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=
> TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/
> rsp:-1:java.net.ConnectException: Connection refused
>
> 2018-01-04 07:35:01.497 ERROR (updateExecutor-2-thread-9-
> processing-http:////sr02:8983//solr//S8IYAB_shard1_replica2
> x:S8IYAB_shard1_replica1 r:core_node2 n:sr01:8983_solr s:shard1 c:S8IYAB)
> [c:S8IYAB s:shard1 r:core_node2 x:S8IYAB_shard1_replica1]
> o.a.s.u.StreamingSolrClients error
> java.net.ConnectException: Connection refused
> ………
>
>
> 2018-01-04 07:35:12.544 ERROR (qtp127618319-21) [c:S8IYAB s:shard1
> r:core_node2 x:S8IYAB_shard1_replica1] o.a.s.u.SolrCmdDistributor
> forwarding update to http://sr02:8983/solr/S8IYAB_shard1_replica2/ failed
> - retrying ... retries: 25 add{,id=0DODD:S8IYAB-
> 13E07D32C2474770A4E6A88EFA0E84F6} params:update.distrib=
> TOLEADER&distrib.from=http://sr01:8983/solr/S8IYAB_shard1_replica1/
> rsp:-1:java.net.ConnectException: Connection refused
> ……..
>
> 2018-01-05 03:54:15.506 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer
> Overseer (id=98943164658291709-sr01:8983_solr-n_0000000071) closing
> 2018-01-05 03:54:15.507 INFO (OverseerStateUpdate-
> 98943164658291709-sr01:8983_solr-n_0000000071) [ ] o.a.s.c.Overseer
> Overseer Loop exiting : sr01:8983_solr
> 2018-01-05 03:54:17.013 WARN
> (zkCallback-4-thread-20-processing-n:sr01:8983_solr)
> [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot
> talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
> 2018-01-05 03:54:17.013 INFO (ShutdownMonitor) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
>
>
> ***********************************************************
> CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the
> exclusive and confidential use of the intended recipient and may constitute
> non-public information. If you received this e-mail in error, disclosing,
> copying, distributing or taking any action in reliance of this e-mail is
> strictly prohibited and may be unlawful. Instead, please notify us
> immediately by return e-mail and promptly delete this message and its
> attachments from your computer system. We do not waive any work product or
> other applicable legal privilege(s) by the transmission of this message.
> ***********************************************************
>
