Hi, I figured out when this issue happens, by observing the patterns with a small set of data.
SPLITSHARD issues a soft-commit (with some of the files still not fully written to disk). If I restart SOLR without issuing an explicit <commit />, the index directory is not fully written and the process is killed. During next restart, the incomplete index is set to 0 records and cleaned up. I think we should update the documentations to let users know that they need to issue a hard <commit /> immediately after a SPLITSHARD operation. I will mention this in the JIRA case I opened for this - https://issues.apache.org/jira/browse/SOLR-11287. Thanks for the help! Thanks, Vanniarajan On Fri, Aug 25, 2017 at 5:29 AM, Erick Erickson <erickerick...@gmail.com> wrote: > This is strange. You can use the "bin/solr zk cp..." commands to bring > the collection state.json down to your local machine and do anything > you want to do with it, including marking the old shard as "active" > and removing the entries for the sub-shards. It's just a text file > after all. I'd also delete the sub-shard directories from my hard > disks. > > Do this _very_ cautiously! And I'd stop all my Solr nodes before doing > anything with it. And copy the original index somewhere for > safekeeping. > > So, let's assume this gets your data back and you try the SPLITSHARD > again. Once that's complete, I'd check the directories of the > sub-shards. Are they (roughly) half the size of your old shard? If so, > copy them somewhere while you investigate what is deleting the data > directories, Solr should absolutely not be doing that. My guess is > it's something external to Solr that wiped them somehow, but how I > have no clue. > > Best, > Erick > > On Thu, Aug 24, 2017 at 1:23 PM, Vannia Rajan <van...@knackforge.com> > wrote: > > Binoy, > > > > I don't see anything wrong with the logs. The newly split shards are up, > > with the parent set to inactive. But before restart, the new shards had > all > > of the data as in parent. After a restart, it wiped off to 0. I checked > the > > data directory, the data is completely gone. > > > > Below, I attach the log messages that happened during the splitshard > > operation completion & when the next restart of SOLR was done. > > > > Collection = COLLECTION > > Shard that lost data = shard2 (xxx.xx.xx.xxx:8983_solr) > > > > *Log messages at the end of splitshard operation:* > > > >> INFO - 2017-08-23 21:58:25.642; [ ] > >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Index on shard: > >> xxx.xx.xx.xxx:8983_solr split into two successfully > >> INFO - 2017-08-23 21:58:25.642; [ ] > >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Applying > buffered > >> updates on : COLLECTION_shard2_0_replica1 > >> INFO - 2017-08-23 21:58:25.643; [ ] > >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Applying > buffered > >> updates on : COLLECTION_shard2_1_replica1 > >> INFO - 2017-08-23 21:58:25.731; [ ] > >> org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null > path=/admin/cores > >> params={async=1000487066830891264&qt=/admin/ > cores&name=COLLECTION_shard2_0_replica1&action=REQUESTAPPLYUPDATES&wt= > javabin&version=2} > >> status=0 QTime=43 > >> INFO - 2017-08-23 21:58:25.774; [ ] > >> org.apache.solr.handler.admin.CoreAdminOperation$13; Applying buffered > >> updates on core: COLLECTION_shard2_0_replica1 > >> INFO - 2017-08-23 21:58:25.774; [ ] > >> org.apache.solr.handler.admin.CoreAdminOperation$13; No buffered > updates > >> available. core=COLLECTION_shard2_0_replica1 > >> INFO - 2017-08-23 21:58:25.906; [ ] > >> org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null > path=/admin/cores > >> params={async=1000487066831162147&qt=/admin/ > cores&name=COLLECTION_shard2_1_replica1&action=REQUESTAPPLYUPDATES&wt= > javabin&version=2} > >> status=0 QTime=131 > >> INFO - 2017-08-23 21:58:25.952; [ ] > >> org.apache.solr.handler.admin.CoreAdminOperation$13; Applying buffered > >> updates on core: COLLECTION_shard2_1_replica1 > >> INFO - 2017-08-23 21:58:25.952; [ ] > >> org.apache.solr.handler.admin.CoreAdminOperation$13; No buffered > updates > >> available. core=COLLECTION_shard2_1_replica1 > >> INFO - 2017-08-23 21:58:25.999; [ ] > >> org.apache.solr.handler.admin.CoreAdminOperation$14; Checking request > >> status for : 1000487066831162147 > >> INFO - 2017-08-23 21:58:26.000; [ ] > >> org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null > path=/admin/cores > >> params={qt=/admin/cores&requestid=1000487066831162147& > action=REQUESTSTATUS&wt=javabin&version=2} > >> status=0 QTime=0 > >> INFO - 2017-08-23 21:58:26.001; [ ] > >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Successfully > >> applied buffered updates on : [COLLECTION_shard2_0_replica1, > >> COLLECTION_shard2_1_replica1] > >> INFO - 2017-08-23 21:58:26.002; [ ] > >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Replication > factor > >> is 1 so switching shard states > >> INFO - 2017-08-23 21:58:26.004; [ ] > >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Successfully > >> created all replica shards for all sub-slices [shard2_0, shard2_1] > >> INFO - 2017-08-23 21:58:26.004; [ ] > >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Calling soft > commit > >> to make sub shard updates visible > >> INFO - 2017-08-23 21:58:26.179; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 1, message = { > >> "shard2":"inactive", > >> "collection":"COLLECTION", > >> "shard2_1":"active", > >> "operation":"updateshardstate", > >> "shard2_0":"active"} current state version: 0 > >> INFO - 2017-08-23 21:58:26.180; [ ] > >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state > invoked for > >> collection: COLLECTION with message: { > >> "shard2":"inactive", > >> "collection":"COLLECTION", > >> "shard2_1":"active", > >> "operation":"updateshardstate", > >> "shard2_0":"active"} > >> INFO - 2017-08-23 21:58:26.180; [ ] > >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state shard2 > to > >> inactive > >> INFO - 2017-08-23 21:58:26.180; [ ] > >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state > shard2_1 to > >> active > >> INFO - 2017-08-23 21:58:26.180; [ ] > >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state > shard2_0 to > >> active > >> INFO - 2017-08-23 21:58:26.226; [ ] > >> org.apache.solr.cloud.overseer.ZkStateWriter; going to > update_collection > >> /collections/COLLECTION/state.json version: 2689 > >> INFO - 2017-08-23 21:58:26.271; [ ] > >> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster > state > >> change: [WatchedEvent state:SyncConnected type:NodeDataChanged > >> path:/collections/COLLECTION/state.json] for collection [COLLECTION] > has > >> occurred - updating... (live nodes size: [8]) > >> INFO - 2017-08-23 21:58:26.271; [ ] > >> org.apache.solr.common.cloud.ZkStateReader; Updating data for > [COLLECTION] > >> from [2689] to [2690] > >> INFO - 2017-08-23 21:58:27.076; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] org.apache.solr.update. > DirectUpdateHandler2; > >> start > >> commit{,optimize=false,openSearcher=true,waitSearcher=true, > expungeDeletes=false,softCommit=true,prepareCommit=false} > >> INFO - 2017-08-23 21:58:27.166; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.update.DirectUpdateHandler2; start > >> commit{,optimize=false,openSearcher=true,waitSearcher=true, > expungeDeletes=false,softCommit=true,prepareCommit=false} > >> INFO - 2017-08-23 21:58:27.255; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrCore; > >> SolrIndexSearcher has not changed - not re-opening: > >> org.apache.solr.search.SolrIndexSearcher > >> INFO - 2017-08-23 21:58:27.255; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.update.DirectUpdateHandler2; start > >> commit{,optimize=false,openSearcher=true,waitSearcher=true, > expungeDeletes=false,softCommit=true,prepareCommit=false} > >> INFO - 2017-08-23 21:58:27.298; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] org.apache.solr.update. > DirectUpdateHandler2; > >> end_commit_flush > >> INFO - 2017-08-23 21:58:27.384; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] > >> org.apache.solr.update.processor.LogUpdateProcessorFactory$ > LogUpdateProcessor; > >> [COLLECTION_shard2_replica1] webapp=/solr path=/update > >> params={update.distrib=FROMLEADER&update.chain=add- > unknown-fields-to-the-schema&waitSearcher=true& > openSearcher=true&commit=true&softCommit=true&distrib.from= > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_ > replica1/&commit_end_point=true&wt=javabin&version=2& > expungeDeletes=false}{commit=} > >> 0 354 > >> INFO - 2017-08-23 21:58:28.754; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.search. > SolrIndexSearcher; > >> Opening Searcher@a4f0979[COLLECTION_shard2_1_replica1] main > >> INFO - 2017-08-23 21:58:28.757; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush > >> INFO - 2017-08-23 21:58:28.758; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core. > QuerySenderListener; > >> QuerySenderListener sending requests to Searcher@a4f0979[COLLECTION_ > shard2_1_replica1] > >> main{ExitableDirectoryReader(UninvertingDirectoryReader( > Uninverting(_0(5.5.1):C1332771) > >> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638) > >> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033) > >> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611) > >> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083) > >> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563) > >> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123) > >> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893) > >> Uninverting(_s(5.5.1):c1969016) Uninverting(_t(5.5.1):c1918535) > >> Uninverting(_u(5.5.1):c1924194) Uninverting(_w(5.5.1):c1660229) > >> Uninverting(_x(5.5.1):c49355) Uninverting(_y(5.5.1):c150056) > >> Uninverting(_z(5.5.1):c63319) Uninverting(_10(5.5.1):c80520) > >> Uninverting(_11(5.5.1):c92203) Uninverting(_12(5.5.1):c96290) > >> Uninverting(_13(5.5.1):c24621) Uninverting(_14(5.5.1):c94896) > >> Uninverting(_15(5.5.1):c8987) Uninverting(_16(5.5.1):c12045) > >> Uninverting(_17(5.5.1):c10943) Uninverting(_18(5.5.1):c7357) > >> Uninverting(_19(5.5.1):c7519) Uninverting(_1a(5.5.1):c2426) > >> Uninverting(_1b(5.5.1):c2700) Uninverting(_1c(5.5.1):c3336) > >> Uninverting(_1d(5.5.1):c5069) Uninverting(_1e(5.5.1):c3707) > >> Uninverting(_1f(5.5.1):c2391) Uninverting(_1g(5.5.1):c1389) > >> Uninverting(_1h(5.5.1):c37) Uninverting(_1i(5.5.1):c42) > >> Uninverting(_1j(5.5.1):c52)))} > >> INFO - 2017-08-23 21:58:28.804; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core. > QuerySenderListener; > >> QuerySenderListener done. > >> INFO - 2017-08-23 21:58:28.804; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.SolrCore; > >> [COLLECTION_shard2_1_replica1] Registered new searcher Searcher@a4f0979 > [COLLECTION_shard2_1_replica1] > >> main{ExitableDirectoryReader(UninvertingDirectoryReader( > Uninverting(_0(5.5.1):C1332771) > >> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638) > >> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033) > >> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611) > >> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083) > >> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563) > >> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123) > >> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893) > >> Uninverting(_s(5.5.1):c1969016) Uninverting(_t(5.5.1):c1918535) > >> Uninverting(_u(5.5.1):c1924194) Uninverting(_w(5.5.1):c1660229) > >> Uninverting(_x(5.5.1):c49355) Uninverting(_y(5.5.1):c150056) > >> Uninverting(_z(5.5.1):c63319) Uninverting(_10(5.5.1):c80520) > >> Uninverting(_11(5.5.1):c92203) Uninverting(_12(5.5.1):c96290) > >> Uninverting(_13(5.5.1):c24621) Uninverting(_14(5.5.1):c94896) > >> Uninverting(_15(5.5.1):c8987) Uninverting(_16(5.5.1):c12045) > >> Uninverting(_17(5.5.1):c10943) Uninverting(_18(5.5.1):c7357) > >> Uninverting(_19(5.5.1):c7519) Uninverting(_1a(5.5.1):c2426) > >> Uninverting(_1b(5.5.1):c2700) Uninverting(_1c(5.5.1):c3336) > >> Uninverting(_1d(5.5.1):c5069) Uninverting(_1e(5.5.1):c3707) > >> Uninverting(_1f(5.5.1):c2391) Uninverting(_1g(5.5.1):c1389) > >> Uninverting(_1h(5.5.1):c37) Uninverting(_1i(5.5.1):c42) > >> Uninverting(_1j(5.5.1):c52)))} > >> INFO - 2017-08-23 21:58:28.804; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.update.processor.LogUpdateProcessorFactory$ > LogUpdateProcessor; > >> [COLLECTION_shard2_1_replica1] webapp=/solr path=/update > >> params={update.distrib=FROMLEADER&update.chain=add- > unknown-fields-to-the-schema&waitSearcher=true& > openSearcher=true&commit=true&softCommit=true&distrib.from= > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_ > replica1/&commit_end_point=true&wt=javabin&version=2& > expungeDeletes=false}{commit=} > >> 0 1728 > >> INFO - 2017-08-23 21:58:37.739; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.search. > SolrIndexSearcher; > >> Opening Searcher@6c639fa[COLLECTION_shard2_0_replica1] main > >> INFO - 2017-08-23 21:58:37.790; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush > >> INFO - 2017-08-23 21:58:37.791; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core. > QuerySenderListener; > >> QuerySenderListener sending requests to Searcher@6c639fa[COLLECTION_ > shard2_0_replica1] > >> main{ExitableDirectoryReader(UninvertingDirectoryReader( > Uninverting(_0(5.5.1):C1334281) > >> Uninverting(_1(5.5.1):C1289239) Uninverting(_l(5.5.1):C3935124) > >> Uninverting(_r(5.5.1):C4462687) Uninverting(_4(5.5.1):C1562371) > >> Uninverting(_f(5.5.1):C3882129) Uninverting(_i(5.5.1):C3961582) > >> Uninverting(_o(5.5.1):C3940869) Uninverting(_c(5.5.1):C3831140) > >> Uninverting(_g(5.5.1):c1143897) Uninverting(_j(5.5.1):c1145692) > >> Uninverting(_m(5.5.1):c1156140) Uninverting(_n(5.5.1):c1599534) > >> Uninverting(_p(5.5.1):c1856273) Uninverting(_q(5.5.1):c1448664) > >> Uninverting(_s(5.5.1):c1971801) Uninverting(_v(5.5.1):c3849462) > >> Uninverting(_w(5.5.1):c1658703) Uninverting(_x(5.5.1):c50020) > >> Uninverting(_y(5.5.1):c150398) Uninverting(_z(5.5.1):c63126) > >> Uninverting(_10(5.5.1):c80664) Uninverting(_11(5.5.1):c92022) > >> Uninverting(_12(5.5.1):c96419) Uninverting(_13(5.5.1):c24559) > >> Uninverting(_14(5.5.1):c94825) Uninverting(_15(5.5.1):c8860) > >> Uninverting(_16(5.5.1):c11971) Uninverting(_17(5.5.1):c10952) > >> Uninverting(_18(5.5.1):c7432) Uninverting(_19(5.5.1):c7526) > >> Uninverting(_1a(5.5.1):c2343) Uninverting(_1b(5.5.1):c2676) > >> Uninverting(_1c(5.5.1):c3202) Uninverting(_1d(5.5.1):c4974) > >> Uninverting(_1e(5.5.1):c3736) Uninverting(_1f(5.5.1):c2345) > >> Uninverting(_1g(5.5.1):c1385) Uninverting(_1h(5.5.1):c43) > >> Uninverting(_1i(5.5.1):c39) Uninverting(_1j(5.5.1):c42)))} > >> INFO - 2017-08-23 21:58:37.836; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core. > QuerySenderListener; > >> QuerySenderListener done. > >> INFO - 2017-08-23 21:58:37.836; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrCore; > >> [COLLECTION_shard2_0_replica1] Registered new searcher Searcher@6c639fa > [COLLECTION_shard2_0_replica1] > >> main{ExitableDirectoryReader(UninvertingDirectoryReader( > Uninverting(_0(5.5.1):C1334281) > >> Uninverting(_1(5.5.1):C1289239) Uninverting(_l(5.5.1):C3935124) > >> Uninverting(_r(5.5.1):C4462687) Uninverting(_4(5.5.1):C1562371) > >> Uninverting(_f(5.5.1):C3882129) Uninverting(_i(5.5.1):C3961582) > >> Uninverting(_o(5.5.1):C3940869) Uninverting(_c(5.5.1):C3831140) > >> Uninverting(_g(5.5.1):c1143897) Uninverting(_j(5.5.1):c1145692) > >> Uninverting(_m(5.5.1):c1156140) Uninverting(_n(5.5.1):c1599534) > >> Uninverting(_p(5.5.1):c1856273) Uninverting(_q(5.5.1):c1448664) > >> Uninverting(_s(5.5.1):c1971801) Uninverting(_v(5.5.1):c3849462) > >> Uninverting(_w(5.5.1):c1658703) Uninverting(_x(5.5.1):c50020) > >> Uninverting(_y(5.5.1):c150398) Uninverting(_z(5.5.1):c63126) > >> Uninverting(_10(5.5.1):c80664) Uninverting(_11(5.5.1):c92022) > >> Uninverting(_12(5.5.1):c96419) Uninverting(_13(5.5.1):c24559) > >> Uninverting(_14(5.5.1):c94825) Uninverting(_15(5.5.1):c8860) > >> Uninverting(_16(5.5.1):c11971) Uninverting(_17(5.5.1):c10952) > >> Uninverting(_18(5.5.1):c7432) Uninverting(_19(5.5.1):c7526) > >> Uninverting(_1a(5.5.1):c2343) Uninverting(_1b(5.5.1):c2676) > >> Uninverting(_1c(5.5.1):c3202) Uninverting(_1d(5.5.1):c4974) > >> Uninverting(_1e(5.5.1):c3736) Uninverting(_1f(5.5.1):c2345) > >> Uninverting(_1g(5.5.1):c1385) Uninverting(_1h(5.5.1):c43) > >> Uninverting(_1i(5.5.1):c39) Uninverting(_1j(5.5.1):c42)))} > >> INFO - 2017-08-23 21:58:37.836; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.update.processor.LogUpdateProcessorFactory$ > LogUpdateProcessor; > >> [COLLECTION_shard2_0_replica1] webapp=/solr path=/update > >> params={update.distrib=FROMLEADER&update.chain=add- > unknown-fields-to-the-schema&waitSearcher=true& > openSearcher=true&commit=true&softCommit=true&distrib.from= > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_ > replica1/&commit_end_point=true&wt=javabin&version=2& > expungeDeletes=false}{commit=} > >> 0 10806 > >> INFO - 2017-08-23 21:58:38.055; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] > >> org.apache.solr.update.processor.LogUpdateProcessorFactory$ > LogUpdateProcessor; > >> [COLLECTION_shard2_replica1] webapp=/solr path=/update > >> params={waitSearcher=true&commit=true&softCommit=true& > wt=javabin&version=2}{commit=} > >> 0 11828 > >> INFO - 2017-08-23 21:58:38.105; [ ] > >> org.apache.solr.common.cloud.SolrZkClient; makePath: > >> /overseer/collection-map-completed/mn-1000 > >> INFO - 2017-08-23 21:58:38.152; [ ] > >> org.apache.solr.cloud.OverseerTaskQueue; Response ZK path: > >> /overseer/collection-queue-work/qnr-0000000025 doesn't exist. > Requestor > >> may have disconnected from ZooKeeper > >> INFO - 2017-08-23 21:58:38.152; [ ] > >> org.apache.solr.cloud.OverseerTaskProcessor$Runner; Overseer Collection > >> Message Handler: Message id:/overseer/collection-queue- > work/qn-0000000025 > >> complete, > >> response:{success={xxx.xx.xx.xxx:8983_solr={responseHeader= > {status=0,QTime=1}},xxx.xx.xx.xxx:8983_solr={responseHeader= > {status=0,QTime=1}},xxx.xx.xx.xxx:8983_solr={responseHeader= > {status=0,QTime=1}},xxx.xx.xx.xxx:8983_solr={responseHeader= > {status=0,QTime=0}},xxx.xx.xx.xxx:8983_solr={responseHeader= > {status=0,QTime=0}},xxx.xx.xx.xxx:8983_solr={responseHeader= > {status=0,QTime=43}},xxx.xx.xx.xxx:8983_solr={responseHeader={status=0, > QTime=131}}, > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_ > replica1/={responseHeader={status=0,QTime=11871}}},1000455258538499159={ > responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId: > >> 1000455258538499159 webapp=null path=/admin/cores > >> params={async=1000455258538499159&qt=/admin/ > cores&collection.configName=COLLECTION&name=COLLECTION_ > shard2_0_replica1&action=CREATE&collection=COLLECTION& > shard=shard2_0&wt=javabin&version=2} > >> status=0 > >> QTime=1},1000455261568273501={responseHeader={status=0, > QTime=0},STATUS=completed,Response=TaskId: > >> 1000455261568273501 webapp=null path=/admin/cores > >> params={async=1000455261568273501&qt=/admin/ > cores&collection.configName=COLLECTION&name=COLLECTION_ > shard2_1_replica1&action=CREATE&collection=COLLECTION& > shard=shard2_1&wt=javabin&version=2} > >> status=0 > >> QTime=1},1000455263597970918={responseHeader={status=0, > QTime=1},STATUS=completed,Response=TaskId: > >> 1000455263597970918 webapp=null path=/admin/cores > >> params={nodeName=xxx.xx.xx.xxx:8983_solr&core=COLLECTION_ > shard2_1_replica1&async=1000455263597970918&qt=/admin/ > cores&coreNodeName=core_node6&action=PREPRECOVERY&checkLive= > true&state=active&onlyIfLeader=true&wt=javabin&version=2} > >> status=0 > >> QTime=1},1000455263617138064={responseHeader={status=0, > QTime=0},STATUS=completed,Response=TaskId: > >> 1000455263617138064 webapp=null path=/admin/cores > >> params={core=COLLECTION_shard2_replica1&async= > 1000455263617138064&qt=/admin/cores&action=SPLIT&targetCore= > COLLECTION_shard2_0_replica1&targetCore=COLLECTION_shard2_ > 1_replica1&wt=javabin&version=2} > >> status=0 > >> QTime=0},1000487066831162147={responseHeader={status=0, > QTime=0},STATUS=completed,Response=TaskId: > >> 1000487066831162147 webapp=null path=/admin/cores > >> params={async=1000487066831162147&qt=/admin/ > cores&name=COLLECTION_shard2_1_replica1&action=REQUESTAPPLYUPDATES&wt= > javabin&version=2} > >> status=0 QTime=131}} > >> INFO - 2017-08-23 21:58:38.156; [ ] > >> org.apache.solr.common.cloud.SolrZkClient; makePath: > >> /overseer/collection-map-running/mn-1001 > >> INFO - 2017-08-23 21:58:38.157; [ ] > >> org.apache.solr.cloud.OverseerTaskProcessor; Overseer Collection > Message > >> Handler: Get the message id:/overseer/collection-queue- > work/qn-0000000026 > >> message:{ > >> "collection":"COLLECTION", > >> "shard":"shard3", > >> "async":"1001", > >> "operation":"splitshard"} > >> INFO - 2017-08-23 21:58:38.202; [ ] > >> org.apache.solr.cloud.OverseerCollectionMessageHandler; > >> OverseerCollectionMessageHandler.processMessage : splitshard , { > >> "collection":"COLLECTION", > >> "shard":"shard3", > >> "async":"1001", > >> "operation":"splitshard"} > >> INFO - 2017-08-23 21:58:38.246; [ ] > >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Split shard > invoked > >> ............ > >> ............ > >> INFO - 2017-08-24 07:41:18.387; [ ] > >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state shard3 > to > >> inactive > >> INFO - 2017-08-24 07:41:18.387; [ ] > >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state > shard3_0 to > >> active > >> INFO - 2017-08-24 07:41:18.387; [ ] > >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state > shard3_1 to > >> active > >> INFO - 2017-08-24 07:41:18.391; [ ] > >> org.apache.solr.cloud.overseer.ZkStateWriter; going to > update_collection > >> /collections/COLLECTION/state.json > >> version: 2700 > >> INFO - 2017-08-24 07:41:18.395; [ ] > >> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster > state > >> change: [WatchedEvent state:Syn > >> cConnected type:NodeDataChanged path:/collections/COLLECTION/ > state.json] > >> for collection [COLLECTION] has occurred - updating... (live nodes > size: [8] > >> ) > >> INFO - 2017-08-24 07:41:18.395; [ ] > >> org.apache.solr.common.cloud.ZkStateReader; Updating data for > [COLLECTION] > >> from [2700] to [2701] > >> INFO - 2017-08-24 07:41:38.169; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] org.apache.solr.update. > DirectUpdateHandler2; > >> start > >> commit{,optimize=false,openSearcher=true,waitSearcher=true, > expungeDeletes=false,softCommit=true,prepareCommit=false} > >> INFO - 2017-08-24 07:41:38.169; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.update.DirectUpdateHandler2; start > >> commit{,optimize=false,openSearcher=true,waitSearcher=true, > expungeDeletes=false,softCommit=true,prepareCommit=false} > >> INFO - 2017-08-24 07:41:38.253; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.update.DirectUpdateHandler2; start > >> commit{,optimize=false,openSearcher=true,waitSearcher=true, > expungeDeletes=false,softCommit=true,prepareCommit=false} > >> INFO - 2017-08-24 07:41:38.793; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrCore; > >> SolrIndexSearcher has not changed - not re-opening: > >> org.apache.solr.search.SolrIndexSearcher > >> INFO - 2017-08-24 07:41:38.793; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrCore; > >> SolrIndexSearcher has not changed - not re-opening: > >> org.apache.solr.search.SolrIndexSearcher > >> INFO - 2017-08-24 07:41:38.837; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush > >> INFO - 2017-08-24 07:41:38.976; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] org.apache.solr.update. > DirectUpdateHandler2; > >> end_commit_flush > >> INFO - 2017-08-24 07:41:38.976; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.update.processor.LogUpdateProc > >> essorFactory$LogUpdateProcessor; [COLLECTION_shard2_0_replica1] > >> webapp=/solr path=/update > >> params={update.distrib=FROMLEADER&update.chain=add-unkn > >> > >> own-fields-to-the-schema&waitSearcher=true& > openSearcher=true&commit=true&softCommit=true&distrib.from= > >> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_sh > >> ard3_replica1/&commit_end_point=true&wt=javabin&version= > 2&expungeDeletes=false}{commit=} > >> 0 807 > >> INFO - 2017-08-24 07:41:38.976; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] > >> org.apache.solr.update.processor.LogUpdateProcesso > >> rFactory$LogUpdateProcessor; [COLLECTION_shard2_replica1] webapp=/solr > >> path=/update params={update.distrib=FROMLEADER&update.chain=add- > unknown-fi > >> > >> elds-to-the-schema&waitSearcher=true&openSearcher=true&commit=true& > softCommit=true&distrib.from= > >> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_shard3_r > >> eplica1/&commit_end_point=true&wt=javabin&version=2& > expungeDeletes=false}{commit=} > >> 0 857 > >> INFO - 2017-08-24 07:41:53.455; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.search. > SolrIndexSearcher; > >> Open > >> ing Searcher@2b1e0336[COLLECTION_shard2_1_replica1] main > >> INFO - 2017-08-24 07:41:53.503; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush > >> INFO - 2017-08-24 07:41:53.503; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core. > QuerySenderListener; > >> QuerySenderListener sending requests to Searcher@2b1e0336[COLLECTION_ > shard2_1_replica1] > >> main{ExitableDirectoryReader(UninvertingDirectoryReader( > Uninverting(_0(5.5.1):C1332771) > >> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638) > >> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033) > >> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611) > >> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083) > >> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563) > >> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123) > >> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893) > >> Uninverting(_s(5.5.1):c1969016) Uninverting(_v(5.5.1):c3842729) > >> Uninverting(_w(5.5.1):c1660229) Uninverting(_x(5.5.1):c49355) > >> Uninverting(_y(5.5.1):c150056) Uninverting(_z(5.5.1):c63319) > >> Uninverting(_10(5.5.1):c80520) Uninverting(_11(5.5.1):c92203) > >> Uninverting(_12(5.5.1):c96290) Uninverting(_13(5.5.1):c24621) > >> Uninverting(_14(5.5.1):c94896) Uninverting(_15(5.5.1):c8987) > >> Uninverting(_16(5.5.1):c12045) Uninverting(_17(5.5.1):c10943) > >> Uninverting(_18(5.5.1):c7357) Uninverting(_19(5.5.1):c7519) > >> Uninverting(_1a(5.5.1):c2426) Uninverting(_1b(5.5.1):c2700) > >> Uninverting(_1c(5.5.1):c3336) Uninverting(_1d(5.5.1):c5069) > >> Uninverting(_1e(5.5.1):c3707) Uninverting(_1f(5.5.1):c2391) > >> Uninverting(_1g(5.5.1):c1389) Uninverting(_1h(5.5.1):c37) > >> Uninverting(_1i(5.5.1):c42) Uninverting(_1j(5.5.1):c52)))} > >> INFO - 2017-08-24 07:41:53.547; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core. > QuerySenderListener; > >> QuerySenderListener done. > >> INFO - 2017-08-24 07:41:53.547; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.SolrCore; > >> [COLLECTION_shard2_1_replica1] Registered new searcher Searcher@2b1e0336 > [COLLECTION_shard2_1_replica1] > >> main{ExitableDirectoryReader(UninvertingDirectoryReader( > Uninverting(_0(5.5.1):C1332771) > >> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638) > >> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033) > >> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611) > >> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083) > >> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563) > >> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123) > >> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893) > >> Uninverting(_s(5.5.1):c1969016) Uninverting(_v(5.5.1):c3842729) > >> Uninverting(_w(5.5.1):c1660229) Uninverting(_x(5.5.1):c49355) > >> Uninverting(_y(5.5.1):c150056) Uninverting(_z(5.5.1):c63319) > >> Uninverting(_10(5.5.1):c80520) Uninverting(_11(5.5.1):c92203) > >> Uninverting(_12(5.5.1):c96290) Uninverting(_13(5.5.1):c24621) > >> Uninverting(_14(5.5.1):c94896) Uninverting(_15(5.5.1):c8987) > >> Uninverting(_16(5.5.1):c12045) Uninverting(_17(5.5.1):c10943) > >> Uninverting(_18(5.5.1):c7357) Uninverting(_19(5.5.1):c7519) > >> Uninverting(_1a(5.5.1):c2426) Uninverting(_1b(5.5.1):c2700) > >> Uninverting(_1c(5.5.1):c3336) Uninverting(_1d(5.5.1):c5069) > >> Uninverting(_1e(5.5.1):c3707) Uninverting(_1f(5.5.1):c2391) > >> Uninverting(_1g(5.5.1):c1389) Uninverting(_1h(5.5.1):c37) > >> Uninverting(_1i(5.5.1):c42) Uninverting(_1j(5.5.1):c52)))} > >> INFO - 2017-08-24 07:41:56.237; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.update.processor.LogUpdateProcessorFactory$ > LogUpdateProcessor; > >> [COLLECTION_shard2_1_replica1] webapp=/solr path=/update > >> params={update.distrib=FROMLEADER&update.chain=add- > unknown-fields-to-the-schema&waitSearcher=true& > openSearcher=true&commit=true&softCommit=true&distrib.from= > >> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_shard3_ > replica1/&commit_end_point=true&wt=javabin&version=2& > expungeDeletes=false}{commit=} > >> 0 18117 > >> INFO - 2017-08-24 07:42:24.803; [ ] > >> org.apache.solr.common.cloud.SolrZkClient; makePath: > >> /overseer/collection-map-completed/mn-1001 > >> INFO - 2017-08-24 07:42:24.856; [ ] > >> org.apache.solr.cloud.OverseerTaskProcessor$Runner; Overseer Collection > >> Message Handler: Message id:/overseer/collection-queue- > work/qn-0000000026 > >> complete, > >> response:{success={xxx.xx.xx.xxx:8984_solr={responseHeader= > {status=0,QTime=4}},xxx.xx.xx.xxx:8984_solr={responseHeader= > {status=0,QTime=0}},xxx.xx.xx.xxx:8984_solr={responseHeader= > {status=0,QTime=1}},xxx.xx.xx.xxx:8984_solr={responseHeader= > {status=0,QTime=1}},xxx.xx.xx.xxx:8984_solr={responseHeader= > {status=0,QTime=1}},xxx.xx.xx.xxx:8984_solr={responseHeader= > {status=0,QTime=0}},xxx.xx.xx.xxx:8984_solr={responseHeader= > {status=0,QTime=189}}, > >> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_shard3_ > replica1/={responseHeader={status=0,QTime=66236}}},1001487080480985685={ > responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId: > >> 1001487080480985685 webapp=null path=/admin/cores > >> params={async=1001487080480985685&qt=/admin/ > cores&collection.configName=COLLECTION&name=COLLECTION_ > shard3_0_replica1&action=CREATE&collection=COLLECTION& > shard=shard3_0&wt=javabin&version=2} > >> status=0 > >> QTime=4},1001487084675100247={responseHeader={status=0, > QTime=0},STATUS=completed,Response=TaskId: > >> 1001487084675100247 webapp=null path=/admin/cores > >> params={async=1001487084675100247&qt=/admin/ > cores&collection.configName=COLLECTION&name=COLLECTION_ > shard3_1_replica1&action=CREATE&collection=COLLECTION& > shard=shard3_1&wt=javabin&version=2} > >> status=0 > >> QTime=0},1001487086698326060={responseHeader={status=0, > QTime=0},STATUS=completed,Response=TaskId: > >> 1001487086698326060 webapp=null path=/admin/cores > >> params={nodeName=xxx.xx.xx.xxx:8984_solr&core=COLLECTION_ > shard3_1_replica1&async=1001487086698326060&qt=/admin/ > cores&coreNodeName=core_node8&action=PREPRECOVERY&checkLive= > true&state=active&onlyIfLeader=true&wt=javabin&version=2} > >> status=0 > >> QTime=1},1001487086756758564={responseHeader={status=0, > QTime=0},STATUS=completed,Response=TaskId: > >> 1001487086756758564 webapp=null path=/admin/cores > >> params={core=COLLECTION_shard3_replica1&async= > 1001487086756758564&qt=/admin/cores&action=SPLIT&targetCore= > COLLECTION_shard3_0_replica1&targetCore=COLLECTION_shard3_ > 1_replica1&wt=javabin&version=2} > >> status=0 > >> QTime=1},1001522038777271446={responseHeader={status=0, > QTime=0},STATUS=completed,Response=TaskId: > >> 1001522038777271446 webapp=null path=/admin/cores > >> params={async=1001522038777271446&qt=/admin/ > cores&name=COLLECTION_shard3_1_replica1&action=REQUESTAPPLYUPDATES&wt= > javabin&version=2} > >> status=0 QTime=189}} > > > > > > *Log messages during next restart:* > > > >> INFO - 2017-08-24 08:52:37.821; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.handler. > ReplicationHandler; > >> Co > >> mmits will be reserved for 10000 > >> INFO - 2017-08-24 08:52:37.826; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core. > SolrDeletionPolicy; > >> SolrD > >> eletionPolicy.onInit: commits: num=1 > >> commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solrcloud/ > solr-5.5.1/example/cloud/node1/solr/COLLECTION_shard2_ > 0_replica1/data/index > >> lo > >> ckFactory=org.apache.lucene.store.NativeFSLockFactory@725fe252; > >> maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1} > >> INFO - 2017-08-24 08:52:37.826; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core. > SolrDeletionPolicy; > >> newes > >> t commit generation = 1 > >> INFO - 2017-08-24 08:52:37.827; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.search. > SolrIndexSearcher; > >> Open > >> ing Searcher@5c49571c[COLLECTION_shard2_0_replica1] main > >> INFO - 2017-08-24 08:52:37.827; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 1, message = { > >> "core":"COLLECTION-digital-archive_shard1_replica1", > >> "core_node_name":"core_node1", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8983/solr";, > >> "node_name":"xxx.xx.xx.xxx:8983_solr", > >> "numShards":"1", > >> "state":"active", > >> "shard":"shard1", > >> "collection":"COLLECTION-digital-archive", > >> "operation":"state"} current state version: 0 > >> INFO - 2017-08-24 08:52:37.827; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud. > ZkStateReader; > >> Load collection config from: [/collections/COLLECTION] > >> INFO - 2017-08-24 08:52:37.827; [ ] > >> org.apache.solr.cloud.overseer.ReplicaMutator; Update state numShards=1 > >> message={ > >> "core":"COLLECTION-digital-archive_shard1_replica1", > >> "core_node_name":"core_node1", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8983/solr";, > >> "node_name":"xxx.xx.xx.xxx:8983_solr", > >> "numShards":"1", > >> "state":"active", > >> "shard":"shard1", > >> "collection":"COLLECTION-digital-archive", > >> "operation":"state"} > >> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud. > ZkStateReader; > >> path=[/collections/COLLECTION] [configName]=[COLLECTION] specified > config > >> exists in ZooKeeper > >> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage; Setting up ZooKeeper-based > >> storage for the RestManager with znodeBase: /configs/COLLECTION > >> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrDeletionPolicy; > >> SolrDeletionPolicy.onInit: commits: num=1 > >> commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solrdata/ > index > >> lockFactory=org.apache.lucene.store.NativeFSLockFactory@725fe252; > >> maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_33sj,generation= > 144883} > >> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.UpdateLog; > Looking > >> up max value of version field to seed version buckets > >> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2 r:core_node1 > >> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrDeletionPolicy; > >> newest commit generation = 144883 > >> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core. > QuerySenderListener; > >> QuerySenderListener sending requests to Searcher@269e2d59[COLLECTION_ > shard2_1_replica1] > >> main{ExitableDirectoryReader(UninvertingDirectoryReader())} > >> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core. > QuerySenderListener; > >> QuerySenderListener done. > >> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.handler.component.SpellCheckComponent$ > SpellCheckerListener; > >> Loading spell index for spellchecker: default > >> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.VersionInfo; > >> Refreshing highest value of _version_ for 65536 version buckets from > index > >> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.SolrCore; > >> [COLLECTION_shard2_1_replica1] Registered new searcher Searcher@269e2d59 > [COLLECTION_shard2_1_replica1] > >> main{ExitableDirectoryReader(UninvertingDirectoryReader())} > >> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.VersionInfo; No > >> terms found for _version_, cannot seed version bucket highest value from > >> index > >> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; > Configured > >> ZooKeeperStorageIO with znodeBase: /configs/COLLECTION > >> INFO - 2017-08-24 08:52:37.830; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.RestManager; > >> Initializing RestManager with initArgs: {} > >> INFO - 2017-08-24 08:52:37.830; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.UpdateLog; > Could not > >> find max version in index or recent updates, using new clock > >> 1576601919506350080 > >> INFO - 2017-08-24 08:52:37.830; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage; Reading _rest_managed.json > >> using ZooKeeperStorageIO:path=/configs/COLLECTION > >> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; Read 33 > >> bytes from znode /configs/COLLECTION/_rest_managed.json > >> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage; Loaded LinkedHashMap at > path > >> _rest_managed.json using ZooKeeperStorageIO:path=/configs/COLLECTION > >> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource; > >> Loaded initArgs {} for /rest/managed > >> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.RestManager; > >> Initializing 2 registered ManagedResources > >> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage; Reading > >> _schema_analysis_stopwords_english.json using > >> ZooKeeperStorageIO:path=/configs/COLLECTION > >> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.UpdateLog; Took > >> 3.0ms to seed version buckets with highest version 1576601919506350080 > >> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; > watch > >> zkdir /configs/COLLECTION > >> INFO - 2017-08-24 08:52:37.833; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.CoreContainer; > >> registering core: COLLECTION_shard2_1_replica1 > >> INFO - 2017-08-24 08:52:37.833; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; Read > 450 > >> bytes from znode /configs/COLLECTION/_schema_ > analysis_stopwords_english.json > >> INFO - 2017-08-24 08:52:37.834; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; > >> Register replica - core:COLLECTION_shard2_1_replica1 address: > >> http://xxx.xx.xx.xxx:8983/solr collection:COLLECTION shard:shard2_1 > >> INFO - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage; Loaded LinkedHashMap at > path > >> _schema_analysis_stopwords_english.json using > >> ZooKeeperStorageIO:path=/configs/COLLECTION > >> INFO - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource; > >> Loaded initArgs {ignoreCase=true} for /schema/analysis/stopwords/ > english > >> INFO - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.schema.analysis.ManagedWordSetResource; Loaded 35 > >> words for /schema/analysis/stopwords/english > >> INFO - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource; > >> Notified 1 observers of /schema/analysis/stopwords/english > >> INFO - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage; Reading > >> _schema_analysis_synonyms_english.json using > >> ZooKeeperStorageIO:path=/configs/COLLECTION > >> INFO - 2017-08-24 08:52:37.837; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.LeaderElector; > Joined > >> leadership election with path: > >> /collections/COLLECTION/leader_elect/shard2_1/ > election/242652799835373569-core_node6-n_0000000003 > >> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; Read > 172 > >> bytes from znode /configs/COLLECTION/_schema_ > analysis_synonyms_english.json > >> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.ManagedResourceStorage; Loaded LinkedHashMap at > path > >> _schema_analysis_synonyms_english.json using > >> ZooKeeperStorageIO:path=/configs/COLLECTION > >> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader > >> process for shard=shard2_1 and weAreReplacement=false and > >> leaderVoteWait=180000 > >> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource; > >> Loaded initArgs {ignoreCase=true,format=solr} for > >> /schema/analysis/synonyms/english > >> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$ > SynonymManager; > >> Loaded 3 synonym mappings for /schema/analysis/synonyms/english > >> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource; > >> Notified 1 observers of /schema/analysis/synonyms/english > >> INFO - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.handler.component.SpellCheckComponent; Initializing > spell > >> checkers > >> INFO - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.spelling.DirectSolrSpellChecker; init: > >> {name=default,field=_text_,classname=solr.DirectSolrSpellChecker, > distanceMeasure=internal,accuracy=0.5,maxEdits=2, > minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01} > >> INFO - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.handler.component.SpellCheckComponent; No > queryConverter > >> defined, using default converter > >> INFO - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud. > ZkStateReader; > >> Load collection config from: [/collections/COLLECTION] > >> INFO - 2017-08-24 08:52:37.840; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud. > ZkStateReader; > >> path=[/collections/COLLECTION] [configName]=[COLLECTION] specified > config > >> exists in ZooKeeper > >> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas > found to > >> continue. > >> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new > leader - > >> try and sync > >> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.SyncStrategy; > Sync > >> replicas to http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_ > replica1/ > >> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.SyncStrategy; > Sync > >> Success - now sync replicas to me > >> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.handler. > ReplicationHandler; > >> Commits will be reserved for 10000 > >> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.SyncStrategy; > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ has no > >> replicas > >> INFO - 2017-08-24 08:52:37.844; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 1, message = { > >> "operation":"leader", > >> "shard":"shard2_1", > >> "collection":"COLLECTION"} current state version: 0 > >> INFO - 2017-08-24 08:52:37.844; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.cloud.ShardLeaderElectionContextBase$1; Creating leader > >> registration node /collections/COLLECTION/leaders/shard2_1/leader after > >> winning as > >> /collections/COLLECTION/leader_elect/shard2_1/ > election/242652799835373569-core_node6-n_0000000003 > >> INFO - 2017-08-24 08:52:37.847; [ ] > >> org.apache.solr.cloud.overseer.ZkStateWriter; going to > update_collection > >> /collections/COLLECTION-digital-archive/state.json version: 902 > >> INFO - 2017-08-24 08:52:37.850; [ ] > >> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster > state > >> change: [WatchedEvent state:SyncConnected type:NodeDataChanged > >> path:/collections/COLLECTION-digital-archive/state.json] for collection > >> [COLLECTION-digital-archive] has occurred - updating... (live nodes > size: > >> [1]) > >> INFO - 2017-08-24 08:52:37.852; [ ] > >> org.apache.solr.common.cloud.ZkStateReader; Updating data for > >> [COLLECTION-digital-archive] from [902] to [903] > >> INFO - 2017-08-24 08:52:37.852; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] > >> org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader: > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ shard2_1 > >> INFO - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core. > QuerySenderListener; > >> QuerySenderListener sending requests to Searcher@5c49571c[COLLECTION_ > shard2_0_replica1] > >> main{ExitableDirectoryReader(UninvertingDirectoryReader())} > >> INFO - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; We > are > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ and leader > >> is http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ > >> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog; > Looking > >> up max value of version field to seed version buckets > >> INFO - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core. > QuerySenderListener; > >> QuerySenderListener done. > >> INFO - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; We > are > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ and leader > >> is http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ > >> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog; > Looking > >> up max value of version field to seed version buckets > >> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.handler.component.SpellCheckComponent$ > SpellCheckerListener; > >> Loading spell index for spellchecker: default > >> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.VersionInfo; > >> Refreshing highest value of _version_ for 65536 version buckets from > index > >> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.VersionInfo; No > >> terms found for _version_, cannot seed version bucket highest value from > >> index > >> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; No > >> LogReplay needed for core=COLLECTION_shard2_1_replica1 baseURL= > >> http://xxx.xx.xx.xxx:8983/solr > >> INFO - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; I > am > >> the leader, no recovery necessary > >> INFO - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog; > Could not > >> find max version in index or recent updates, using new clock > >> 1576601919533613056 > >> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrCore; > >> [COLLECTION_shard2_0_replica1] Registered new searcher Searcher@5c49571c > [COLLECTION_shard2_0_replica1] > >> main{ExitableDirectoryReader(UninvertingDirectoryReader())} > >> INFO - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; > >> publishing state=active > >> INFO - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_1 r:core_node6 > >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; > >> numShards not found on descriptor - reading it from system property > >> INFO - 2017-08-24 08:52:37.857; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog; Took > >> 2.0ms to seed version buckets with highest version 1576601919533613056 > >> INFO - 2017-08-24 08:52:37.857; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.CoreContainer; > >> registering core: COLLECTION_shard2_0_replica1 > >> INFO - 2017-08-24 08:52:37.858; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; > >> Register replica - core:COLLECTION_shard2_0_replica1 address: > >> http://xxx.xx.xx.xxx:8983/solr collection:COLLECTION shard:shard2_0 > >> INFO - 2017-08-24 08:52:37.862; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.LeaderElector; > Joined > >> leadership election with path: > >> /collections/COLLECTION/leader_elect/shard2_0/ > election/242652799835373569-core_node5-n_0000000003 > >> INFO - 2017-08-24 08:52:37.864; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader > >> process for shard=shard2_0 and weAreReplacement=false and > >> leaderVoteWait=180000 > >> INFO - 2017-08-24 08:52:37.867; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 2, message = { > >> "operation":"leader", > >> "shard":"shard2_1", > >> "collection":"COLLECTION", > >> "base_url":"http://xxx.xx.xx.xxx:8983/solr";, > >> "core":"COLLECTION_shard2_1_replica1", > >> "state":"active"} current state version: 0 > >> INFO - 2017-08-24 08:52:37.871; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas > found to > >> continue. > >> INFO - 2017-08-24 08:52:37.871; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new > leader - > >> try and sync > >> INFO - 2017-08-24 08:52:37.872; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.SyncStrategy; > Sync > >> replicas to http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_ > replica1/ > >> INFO - 2017-08-24 08:52:37.872; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.SyncStrategy; > Sync > >> Success - now sync replicas to me > >> INFO - 2017-08-24 08:52:37.872; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.SyncStrategy; > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/ has no > >> replicas > >> INFO - 2017-08-24 08:52:37.874; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.cloud.ShardLeaderElectionContextBase$1; Creating leader > >> registration node /collections/COLLECTION/leaders/shard2_0/leader after > >> winning as > >> /collections/COLLECTION/leader_elect/shard2_0/ > election/242652799835373569-core_node5-n_0000000003 > >> INFO - 2017-08-24 08:52:37.874; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 2, message = { > >> "core":"COLLECTION_shard2_1_replica1", > >> "core_node_name":"core_node6", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8983/solr";, > >> "node_name":"xxx.xx.xx.xxx:8983_solr", > >> "state":"active", > >> "shard":"shard2_1", > >> "collection":"COLLECTION", > >> "operation":"state"} current state version: 0 > >> INFO - 2017-08-24 08:52:37.876; [ ] > >> org.apache.solr.cloud.overseer.ReplicaMutator; Update state > numShards=null > >> message={ > >> "core":"COLLECTION_shard2_1_replica1", > >> "core_node_name":"core_node6", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8983/solr";, > >> "node_name":"xxx.xx.xx.xxx:8983_solr", > >> "state":"active", > >> "shard":"shard2_1", > >> "collection":"COLLECTION", > >> "operation":"state"} > >> INFO - 2017-08-24 08:52:37.881; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] > >> org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader: > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/ shard2_0 > >> INFO - 2017-08-24 08:52:37.882; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 2, message = { > >> "operation":"leader", > >> "shard":"shard2_0", > >> "collection":"COLLECTION"} current state version: 0 > >> INFO - 2017-08-24 08:52:37.882; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; We > are > >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/ and leader > >> is http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/ > >> INFO - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; No > >> LogReplay needed for core=COLLECTION_shard2_0_replica1 baseURL= > >> http://xxx.xx.xx.xxx:8983/solr > >> INFO - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; I > am > >> the leader, no recovery necessary > >> INFO - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; > >> publishing state=active > >> INFO - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5 > >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; > >> numShards not found on descriptor - reading it from system property > >> INFO - 2017-08-24 08:52:37.888; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 2, message = { > >> "operation":"leader", > >> "shard":"shard2_0", > >> "collection":"COLLECTION", > >> "base_url":"http://xxx.xx.xx.xxx:8983/solr";, > >> "core":"COLLECTION_shard2_0_replica1", > >> "state":"active"} current state version: 0 > >> INFO - 2017-08-24 08:52:37.894; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 1, message = { > >> "core":"COLLECTION_shard2_0_replica1", > >> "core_node_name":"core_node5", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8983/solr";, > >> "node_name":"xxx.xx.xx.xxx:8983_solr", > >> "state":"active", > >> "shard":"shard2_0", > >> "collection":"COLLECTION", > >> "operation":"state"} current state version: 0 > >> INFO - 2017-08-24 08:52:37.894; [ ] > >> org.apache.solr.cloud.overseer.ReplicaMutator; Update state > numShards=null > >> message={ > >> "core":"COLLECTION_shard2_0_replica1", > >> "core_node_name":"core_node5", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8983/solr";, > >> "node_name":"xxx.xx.xx.xxx:8983_solr", > >> "state":"active", > >> "shard":"shard2_0", > >> "collection":"COLLECTION", > >> "operation":"state"} > >> INFO - 2017-08-24 08:52:38.000; [ ] > >> org.apache.solr.cloud.overseer.ZkStateWriter; going to > update_collection > >> /collections/COLLECTION/state.json version: 2721 > >> INFO - 2017-08-24 08:52:38.005; [ ] > >> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster > state > >> change: [WatchedEvent state:SyncConnected type:NodeDataChanged > >> path:/collections/COLLECTION/state.json] for collection [COLLECTION] > has > >> occurred - updating... (live nodes size: [1]) > >> INFO - 2017-08-24 08:52:38.006; [ ] > >> org.apache.solr.common.cloud.ZkStateReader; Updating data for > [COLLECTION] > >> from [2721] to [2722] > >> INFO - 2017-08-24 08:52:40.805; [ ] > >> org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher; A live node > >> change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged > >> path:/live_nodes], has occurred - updating... (live nodes size: [1]) > >> INFO - 2017-08-24 08:52:40.931; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 1, message = { > >> "core":"COLLECTION1_shard4_replica1", > >> "core_node_name":"core_node3", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8984/solr";, > >> "node_name":"xxx.xx.xx.xxx:8984_solr", > >> "numShards":"4", > >> "state":"down", > >> "shard":"shard4", > >> "collection":"COLLECTION1", > >> "operation":"state"} current state version: 0 > >> INFO - 2017-08-24 08:52:40.931; [ ] > >> org.apache.solr.cloud.overseer.ReplicaMutator; Update state numShards=4 > >> message={ > >> "core":"COLLECTION1_shard4_replica1", > >> "core_node_name":"core_node3", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8984/solr";, > >> "node_name":"xxx.xx.xx.xxx:8984_solr", > >> "numShards":"4", > >> "state":"down", > >> "shard":"shard4", > >> "collection":"COLLECTION1", > >> "operation":"state"} > >> INFO - 2017-08-24 08:52:40.934; [ ] > >> org.apache.solr.cloud.overseer.ZkStateWriter; going to > update_collection > >> /collections/COLLECTION1/state.json version: 429 > >> INFO - 2017-08-24 08:52:40.935; [ ] > >> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster > state > >> change: [WatchedEvent state:SyncConnected type:NodeDataChanged > >> path:/collections/COLLECTION1/state.json] for collection [COLLECTION1] > has > >> occurred - updating... (live nodes size: [2]) > >> INFO - 2017-08-24 08:52:40.936; [ ] > >> org.apache.solr.common.cloud.ZkStateReader; Updating data for > [COLLECTION1] > >> from [429] to [430] > >> INFO - 2017-08-24 08:52:40.940; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 4, message = { > >> "core":"COLLECTION_shard3_0_replica1", > >> "core_node_name":"core_node7", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8984/solr";, > >> "node_name":"xxx.xx.xx.xxx:8984_solr", > >> "state":"down", > >> "shard":"shard3_0", > >> "collection":"COLLECTION", > >> "operation":"state"} current state version: 0 > >> INFO - 2017-08-24 08:52:40.940; [ ] > >> org.apache.solr.cloud.overseer.ReplicaMutator; Update state > numShards=null > >> message={ > >> "core":"COLLECTION_shard3_0_replica1", > >> "core_node_name":"core_node7", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8984/solr";, > >> "node_name":"xxx.xx.xx.xxx:8984_solr", > >> "state":"down", > >> "shard":"shard3_0", > >> "collection":"COLLECTION", > >> "operation":"state"} > >> INFO - 2017-08-24 08:52:40.946; [ ] > >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage: > >> queueSize: 0, message = { > >> "core":"COLLECTION_sandbox_shard4_replica1", > >> "core_node_name":"core_node4", > >> "roles":null, > >> "base_url":"http://xxx.xx.xx.xxx:8984/solr";, > >> "node_name":"xxx.xx.xx.xxx:8984_solr", > >> "numShards":"4", > >> "state":"down", > >> "shard":"shard4", > > > > > > > > Thanks, > > Vanniarajan > > > > On Thu, Aug 24, 2017 at 8:36 PM, Binoy Dalal <binoydala...@gmail.com> > wrote: > > > >> Vanniarajan, > >> Do you see errors in your solr logs when you reboot? If so paste them > here. > >> Are both the shards up? > >> Have you checked the data directory for both and confirmed that the > data is > >> gone? > >> > >> On Thu 24 Aug, 2017, 18:04 Vannia Rajan <van...@knackforge.com> wrote: > >> > >> > Hi, > >> > > >> > I'm facing weird issues on using SPLITSHARD on fairly large shards > (150GB > >> > shard size), using SOLR 5.5.1 in cloud mode (4 nodes, 1 shard per > node). > >> > > >> > After issuing SPLITSHARD, I successfully get the sub-shards activated > and > >> > my parent shard set "inactive". The newly split shards counts equal > the > >> > parent's counts and all works well. The issue is when I restart SOLR, > the > >> > newly split shards gets reduced to 71KB and 0 documents. > >> > > >> > Has anyone faced such issues, and do you think this can be a possible > bug > >> > that could be fixed if I upgrade SOLR to 6.6? > >> > > >> > Also, I would like to know if there is a way to "rollback" the > >> SPLITSHARD. > >> > I would like to reactivate my parent shard with all documents intact > and > >> > delete the new sub-shards. I have no idea how I could reactivate a > shard > >> > that is now marked as "inactive". > >> > > >> > Thanks, > >> > Vanniarajan > >> > > >> -- > >> Regards, > >> Binoy Dalal > >> >
