We bounced ZooKeeper nodes one by one but not change Since this is our Prod server (100M+ docs) we don¹t want to have to reindex from scratch (takes 7+ days) So we¹re considering editing /collections/<collection>/state.json via zkcli.sh
Thoughts? -Frank On 1/17/17, 5:49 PM, "Pushkar Raste" <pushkar.ra...@gmail.com> wrote: >Try bouncing the overseer for your cluster. > >On Jan 17, 2017 12:01 PM, "Kelly, Frank" <frank.ke...@here.com> wrote: > >> Solr Version: 5.3.1 >> >> Configuration: 3 shards, 3 replicas each >> >> After running out of heap memory recently (cause unknown) we¹ve been >> successfully restarting nodes to recover. >> >> Finally we did one restart and one of the nodes now says the following >> 2017-01-17 16:57:16.835 ERROR (qtp1395089624-17) >> [c:prod_us-east-1_here_account s:shard3 r:core_node26 >> x:prod_us-east-1_here_account_shard3_replica3] o.a.s.c.SolrCore >> org.apache.solr.common.SolrException: ClusterState says we are the >>leader >> >>(https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.255 >>.6.196%3A8983%2Fsolr%2Fprod_us-east-1_here_account_shard3_replica3&data=0 >>1%7C01%7C%7Ce6ff77fca8124af5cd7c08d43f2b2cf6%7C6d4034cd72254f72b85391feae >>a64919%7C1&sdata=iN%2FH3MxId9b6aKcch%2FKSHYncsM5Ug6QJhRfGJ59G7uo%3D&reser >>ved=0), >> but locally we don't think so. Request came from null >> >> How can we recover from this (for Solr 5.3.1)? >> Is there someway to force a new leader (I know the following feature >> exists but in 5.4.0 >>https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues >>.apache.org%2Fjira%2Fbrowse%2FSOLR-7569&data=01%7C01%7C%7Ce6ff77fca8124af >>5cd7c08d43f2b2cf6%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=O7Ey7mUyuy >>Kv%2FEPQvewITfxw8cmTEpUlV18hXGeIKUI%3D&reserved=0) >> >> Thanks! >> >> -Frank >> >> [image: Description: Macintosh >> >>HD:Users:jerchow:Downloads:Asset_Package_01_160721:HERE_Logo_2016:sRGB:PD >>F:HERE_Logo_2016_POS_sRGB.pdf] >> >> >> >> *Frank Kelly* >> >> *Principal Software Engineer* >> >> >> >> HERE >> >> 5 Wayside Rd, Burlington, MA 01803, USA >> >> *42° 29' 7" N 71° 11' 32" W* >> >> >> [image: Description: >> >>/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Ima >>ges/20160726_HERE_EMail_Signature_360.gif] >> >><https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F360.he >>re.com%2F&data=01%7C01%7C%7Ce6ff77fca8124af5cd7c08d43f2b2cf6%7C6d4034cd72 >>254f72b85391feaea64919%7C1&sdata=wDqPoxFI%2BLRB7kZ%2FVkvGPZnz8vHK4VoeEjcK >>%2FXxTo%2BU%3D&reserved=0> [image: Description: >> >>/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Ima >>ges/20160726_HERE_EMail_Signature_Twitter.gif] >> >><https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.t >>witter.com%2Fhere&data=01%7C01%7C%7Ce6ff77fca8124af5cd7c08d43f2b2cf6%7C6d >>4034cd72254f72b85391feaea64919%7C1&sdata=La8C0bkCRhubUHz6KwLtMnLcv1Txglck >>ua2YKDldM84%3D&reserved=0> [image: Description: >> >>/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Ima >>ges/20160726_HERE_EMail_Signature_FB.gif] >> >><https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.f >>acebook.com%2Fhere&data=01%7C01%7C%7Ce6ff77fca8124af5cd7c08d43f2b2cf6%7C6 >>d4034cd72254f72b85391feaea64919%7C1&sdata=ngYNMYgIra7Kb1FPus5sp1b%2B5EBis >>ws9%2Fsy10K%2B4l9I%3D&reserved=0> [image: Description: >> >>/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Ima >>ges/20160726_HERE_EMail_Signature_IN.gif] >> >><https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.l >>inkedin.com%2Fcompany%2Fheremaps&data=01%7C01%7C%7Ce6ff77fca8124af5cd7c08 >>d43f2b2cf6%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=KqQpmAcp2l2h4R%2B >>slYZGf0%2Brwa8c9lzbzeIK%2B94ifrc%3D&reserved=0> [image: Description: >> >>/Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Ima >>ges/20160726_HERE_EMail_Signature_Insta.gif] >> >><https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.i >>nstagram.com%2Fhere%2F&data=01%7C01%7C%7Ce6ff77fca8124af5cd7c08d43f2b2cf6 >>%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=ys7%2FxZdOGXtlVMFF2%2FBn3At >>pBN0GuOPePmf9sH4j8Dg%3D&reserved=0> >>
