Binoy,
I don't see anything wrong with the logs. The newly split shards are up,
with the parent set to inactive. But before restart, the new shards had all
of the data as in parent. After a restart, it wiped off to 0. I checked the
data directory, the data is completely gone.
Below, I attach the log messages that happened during the splitshard
operation completion & when the next restart of SOLR was done.
Collection = COLLECTION
Shard that lost data = shard2 (xxx.xx.xx.xxx:8983_solr)
*Log messages at the end of splitshard operation:*
> INFO - 2017-08-23 21:58:25.642; [ ]
> org.apache.solr.cloud.OverseerCollectionMessageHandler; Index on shard:
> xxx.xx.xx.xxx:8983_solr split into two successfully
> INFO - 2017-08-23 21:58:25.642; [ ]
> org.apache.solr.cloud.OverseerCollectionMessageHandler; Applying buffered
> updates on : COLLECTION_shard2_0_replica1
> INFO - 2017-08-23 21:58:25.643; [ ]
> org.apache.solr.cloud.OverseerCollectionMessageHandler; Applying buffered
> updates on : COLLECTION_shard2_1_replica1
> INFO - 2017-08-23 21:58:25.731; [ ]
> org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null path=/admin/cores
> params={async=1000487066830891264&qt=/admin/cores&name=COLLECTION_shard2_0_replica1&action=REQUESTAPPLYUPDATES&wt=javabin&version=2}
> status=0 QTime=43
> INFO - 2017-08-23 21:58:25.774; [ ]
> org.apache.solr.handler.admin.CoreAdminOperation$13; Applying buffered
> updates on core: COLLECTION_shard2_0_replica1
> INFO - 2017-08-23 21:58:25.774; [ ]
> org.apache.solr.handler.admin.CoreAdminOperation$13; No buffered updates
> available. core=COLLECTION_shard2_0_replica1
> INFO - 2017-08-23 21:58:25.906; [ ]
> org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null path=/admin/cores
> params={async=1000487066831162147&qt=/admin/cores&name=COLLECTION_shard2_1_replica1&action=REQUESTAPPLYUPDATES&wt=javabin&version=2}
> status=0 QTime=131
> INFO - 2017-08-23 21:58:25.952; [ ]
> org.apache.solr.handler.admin.CoreAdminOperation$13; Applying buffered
> updates on core: COLLECTION_shard2_1_replica1
> INFO - 2017-08-23 21:58:25.952; [ ]
> org.apache.solr.handler.admin.CoreAdminOperation$13; No buffered updates
> available. core=COLLECTION_shard2_1_replica1
> INFO - 2017-08-23 21:58:25.999; [ ]
> org.apache.solr.handler.admin.CoreAdminOperation$14; Checking request
> status for : 1000487066831162147
> INFO - 2017-08-23 21:58:26.000; [ ]
> org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null path=/admin/cores
> params={qt=/admin/cores&requestid=1000487066831162147&action=REQUESTSTATUS&wt=javabin&version=2}
> status=0 QTime=0
> INFO - 2017-08-23 21:58:26.001; [ ]
> org.apache.solr.cloud.OverseerCollectionMessageHandler; Successfully
> applied buffered updates on : [COLLECTION_shard2_0_replica1,
> COLLECTION_shard2_1_replica1]
> INFO - 2017-08-23 21:58:26.002; [ ]
> org.apache.solr.cloud.OverseerCollectionMessageHandler; Replication factor
> is 1 so switching shard states
> INFO - 2017-08-23 21:58:26.004; [ ]
> org.apache.solr.cloud.OverseerCollectionMessageHandler; Successfully
> created all replica shards for all sub-slices [shard2_0, shard2_1]
> INFO - 2017-08-23 21:58:26.004; [ ]
> org.apache.solr.cloud.OverseerCollectionMessageHandler; Calling soft commit
> to make sub shard updates visible
> INFO - 2017-08-23 21:58:26.179; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 1, message = {
> "shard2":"inactive",
> "collection":"COLLECTION",
> "shard2_1":"active",
> "operation":"updateshardstate",
> "shard2_0":"active"} current state version: 0
> INFO - 2017-08-23 21:58:26.180; [ ]
> org.apache.solr.cloud.overseer.SliceMutator; Update shard state invoked for
> collection: COLLECTION with message: {
> "shard2":"inactive",
> "collection":"COLLECTION",
> "shard2_1":"active",
> "operation":"updateshardstate",
> "shard2_0":"active"}
> INFO - 2017-08-23 21:58:26.180; [ ]
> org.apache.solr.cloud.overseer.SliceMutator; Update shard state shard2 to
> inactive
> INFO - 2017-08-23 21:58:26.180; [ ]
> org.apache.solr.cloud.overseer.SliceMutator; Update shard state shard2_1 to
> active
> INFO - 2017-08-23 21:58:26.180; [ ]
> org.apache.solr.cloud.overseer.SliceMutator; Update shard state shard2_0 to
> active
> INFO - 2017-08-23 21:58:26.226; [ ]
> org.apache.solr.cloud.overseer.ZkStateWriter; going to update_collection
> /collections/COLLECTION/state.json version: 2689
> INFO - 2017-08-23 21:58:26.271; [ ]
> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/COLLECTION/state.json] for collection [COLLECTION] has
> occurred - updating... (live nodes size: [8])
> INFO - 2017-08-23 21:58:26.271; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for [COLLECTION]
> from [2689] to [2690]
> INFO - 2017-08-23 21:58:27.076; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1] org.apache.solr.update.DirectUpdateHandler2;
> start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> INFO - 2017-08-23 21:58:27.166; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.update.DirectUpdateHandler2; start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> INFO - 2017-08-23 21:58:27.255; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrCore;
> SolrIndexSearcher has not changed - not re-opening:
> org.apache.solr.search.SolrIndexSearcher
> INFO - 2017-08-23 21:58:27.255; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.update.DirectUpdateHandler2; start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> INFO - 2017-08-23 21:58:27.298; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1] org.apache.solr.update.DirectUpdateHandler2;
> end_commit_flush
> INFO - 2017-08-23 21:58:27.384; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [COLLECTION_shard2_replica1] webapp=/solr path=/update
> params={update.distrib=FROMLEADER&update.chain=add-unknown-fields-to-the-schema&waitSearcher=true&openSearcher=true&commit=true&softCommit=true&distrib.from=
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
> 0 354
> INFO - 2017-08-23 21:58:28.754; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.search.SolrIndexSearcher;
> Opening Searcher@a4f0979[COLLECTION_shard2_1_replica1] main
> INFO - 2017-08-23 21:58:28.757; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
> INFO - 2017-08-23 21:58:28.758; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.QuerySenderListener;
> QuerySenderListener sending requests to
> Searcher@a4f0979[COLLECTION_shard2_1_replica1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.5.1):C1332771)
> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638)
> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033)
> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611)
> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083)
> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563)
> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123)
> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893)
> Uninverting(_s(5.5.1):c1969016) Uninverting(_t(5.5.1):c1918535)
> Uninverting(_u(5.5.1):c1924194) Uninverting(_w(5.5.1):c1660229)
> Uninverting(_x(5.5.1):c49355) Uninverting(_y(5.5.1):c150056)
> Uninverting(_z(5.5.1):c63319) Uninverting(_10(5.5.1):c80520)
> Uninverting(_11(5.5.1):c92203) Uninverting(_12(5.5.1):c96290)
> Uninverting(_13(5.5.1):c24621) Uninverting(_14(5.5.1):c94896)
> Uninverting(_15(5.5.1):c8987) Uninverting(_16(5.5.1):c12045)
> Uninverting(_17(5.5.1):c10943) Uninverting(_18(5.5.1):c7357)
> Uninverting(_19(5.5.1):c7519) Uninverting(_1a(5.5.1):c2426)
> Uninverting(_1b(5.5.1):c2700) Uninverting(_1c(5.5.1):c3336)
> Uninverting(_1d(5.5.1):c5069) Uninverting(_1e(5.5.1):c3707)
> Uninverting(_1f(5.5.1):c2391) Uninverting(_1g(5.5.1):c1389)
> Uninverting(_1h(5.5.1):c37) Uninverting(_1i(5.5.1):c42)
> Uninverting(_1j(5.5.1):c52)))}
> INFO - 2017-08-23 21:58:28.804; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.QuerySenderListener;
> QuerySenderListener done.
> INFO - 2017-08-23 21:58:28.804; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.SolrCore;
> [COLLECTION_shard2_1_replica1] Registered new searcher
> Searcher@a4f0979[COLLECTION_shard2_1_replica1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.5.1):C1332771)
> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638)
> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033)
> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611)
> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083)
> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563)
> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123)
> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893)
> Uninverting(_s(5.5.1):c1969016) Uninverting(_t(5.5.1):c1918535)
> Uninverting(_u(5.5.1):c1924194) Uninverting(_w(5.5.1):c1660229)
> Uninverting(_x(5.5.1):c49355) Uninverting(_y(5.5.1):c150056)
> Uninverting(_z(5.5.1):c63319) Uninverting(_10(5.5.1):c80520)
> Uninverting(_11(5.5.1):c92203) Uninverting(_12(5.5.1):c96290)
> Uninverting(_13(5.5.1):c24621) Uninverting(_14(5.5.1):c94896)
> Uninverting(_15(5.5.1):c8987) Uninverting(_16(5.5.1):c12045)
> Uninverting(_17(5.5.1):c10943) Uninverting(_18(5.5.1):c7357)
> Uninverting(_19(5.5.1):c7519) Uninverting(_1a(5.5.1):c2426)
> Uninverting(_1b(5.5.1):c2700) Uninverting(_1c(5.5.1):c3336)
> Uninverting(_1d(5.5.1):c5069) Uninverting(_1e(5.5.1):c3707)
> Uninverting(_1f(5.5.1):c2391) Uninverting(_1g(5.5.1):c1389)
> Uninverting(_1h(5.5.1):c37) Uninverting(_1i(5.5.1):c42)
> Uninverting(_1j(5.5.1):c52)))}
> INFO - 2017-08-23 21:58:28.804; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [COLLECTION_shard2_1_replica1] webapp=/solr path=/update
> params={update.distrib=FROMLEADER&update.chain=add-unknown-fields-to-the-schema&waitSearcher=true&openSearcher=true&commit=true&softCommit=true&distrib.from=
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
> 0 1728
> INFO - 2017-08-23 21:58:37.739; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.search.SolrIndexSearcher;
> Opening Searcher@6c639fa[COLLECTION_shard2_0_replica1] main
> INFO - 2017-08-23 21:58:37.790; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
> INFO - 2017-08-23 21:58:37.791; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.QuerySenderListener;
> QuerySenderListener sending requests to
> Searcher@6c639fa[COLLECTION_shard2_0_replica1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.5.1):C1334281)
> Uninverting(_1(5.5.1):C1289239) Uninverting(_l(5.5.1):C3935124)
> Uninverting(_r(5.5.1):C4462687) Uninverting(_4(5.5.1):C1562371)
> Uninverting(_f(5.5.1):C3882129) Uninverting(_i(5.5.1):C3961582)
> Uninverting(_o(5.5.1):C3940869) Uninverting(_c(5.5.1):C3831140)
> Uninverting(_g(5.5.1):c1143897) Uninverting(_j(5.5.1):c1145692)
> Uninverting(_m(5.5.1):c1156140) Uninverting(_n(5.5.1):c1599534)
> Uninverting(_p(5.5.1):c1856273) Uninverting(_q(5.5.1):c1448664)
> Uninverting(_s(5.5.1):c1971801) Uninverting(_v(5.5.1):c3849462)
> Uninverting(_w(5.5.1):c1658703) Uninverting(_x(5.5.1):c50020)
> Uninverting(_y(5.5.1):c150398) Uninverting(_z(5.5.1):c63126)
> Uninverting(_10(5.5.1):c80664) Uninverting(_11(5.5.1):c92022)
> Uninverting(_12(5.5.1):c96419) Uninverting(_13(5.5.1):c24559)
> Uninverting(_14(5.5.1):c94825) Uninverting(_15(5.5.1):c8860)
> Uninverting(_16(5.5.1):c11971) Uninverting(_17(5.5.1):c10952)
> Uninverting(_18(5.5.1):c7432) Uninverting(_19(5.5.1):c7526)
> Uninverting(_1a(5.5.1):c2343) Uninverting(_1b(5.5.1):c2676)
> Uninverting(_1c(5.5.1):c3202) Uninverting(_1d(5.5.1):c4974)
> Uninverting(_1e(5.5.1):c3736) Uninverting(_1f(5.5.1):c2345)
> Uninverting(_1g(5.5.1):c1385) Uninverting(_1h(5.5.1):c43)
> Uninverting(_1i(5.5.1):c39) Uninverting(_1j(5.5.1):c42)))}
> INFO - 2017-08-23 21:58:37.836; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.QuerySenderListener;
> QuerySenderListener done.
> INFO - 2017-08-23 21:58:37.836; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrCore;
> [COLLECTION_shard2_0_replica1] Registered new searcher
> Searcher@6c639fa[COLLECTION_shard2_0_replica1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.5.1):C1334281)
> Uninverting(_1(5.5.1):C1289239) Uninverting(_l(5.5.1):C3935124)
> Uninverting(_r(5.5.1):C4462687) Uninverting(_4(5.5.1):C1562371)
> Uninverting(_f(5.5.1):C3882129) Uninverting(_i(5.5.1):C3961582)
> Uninverting(_o(5.5.1):C3940869) Uninverting(_c(5.5.1):C3831140)
> Uninverting(_g(5.5.1):c1143897) Uninverting(_j(5.5.1):c1145692)
> Uninverting(_m(5.5.1):c1156140) Uninverting(_n(5.5.1):c1599534)
> Uninverting(_p(5.5.1):c1856273) Uninverting(_q(5.5.1):c1448664)
> Uninverting(_s(5.5.1):c1971801) Uninverting(_v(5.5.1):c3849462)
> Uninverting(_w(5.5.1):c1658703) Uninverting(_x(5.5.1):c50020)
> Uninverting(_y(5.5.1):c150398) Uninverting(_z(5.5.1):c63126)
> Uninverting(_10(5.5.1):c80664) Uninverting(_11(5.5.1):c92022)
> Uninverting(_12(5.5.1):c96419) Uninverting(_13(5.5.1):c24559)
> Uninverting(_14(5.5.1):c94825) Uninverting(_15(5.5.1):c8860)
> Uninverting(_16(5.5.1):c11971) Uninverting(_17(5.5.1):c10952)
> Uninverting(_18(5.5.1):c7432) Uninverting(_19(5.5.1):c7526)
> Uninverting(_1a(5.5.1):c2343) Uninverting(_1b(5.5.1):c2676)
> Uninverting(_1c(5.5.1):c3202) Uninverting(_1d(5.5.1):c4974)
> Uninverting(_1e(5.5.1):c3736) Uninverting(_1f(5.5.1):c2345)
> Uninverting(_1g(5.5.1):c1385) Uninverting(_1h(5.5.1):c43)
> Uninverting(_1i(5.5.1):c39) Uninverting(_1j(5.5.1):c42)))}
> INFO - 2017-08-23 21:58:37.836; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [COLLECTION_shard2_0_replica1] webapp=/solr path=/update
> params={update.distrib=FROMLEADER&update.chain=add-unknown-fields-to-the-schema&waitSearcher=true&openSearcher=true&commit=true&softCommit=true&distrib.from=
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
> 0 10806
> INFO - 2017-08-23 21:58:38.055; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [COLLECTION_shard2_replica1] webapp=/solr path=/update
> params={waitSearcher=true&commit=true&softCommit=true&wt=javabin&version=2}{commit=}
> 0 11828
> INFO - 2017-08-23 21:58:38.105; [ ]
> org.apache.solr.common.cloud.SolrZkClient; makePath:
> /overseer/collection-map-completed/mn-1000
> INFO - 2017-08-23 21:58:38.152; [ ]
> org.apache.solr.cloud.OverseerTaskQueue; Response ZK path:
> /overseer/collection-queue-work/qnr-0000000025 doesn't exist. Requestor
> may have disconnected from ZooKeeper
> INFO - 2017-08-23 21:58:38.152; [ ]
> org.apache.solr.cloud.OverseerTaskProcessor$Runner; Overseer Collection
> Message Handler: Message id:/overseer/collection-queue-work/qn-0000000025
> complete,
> response:{success={xxx.xx.xx.xxx:8983_solr={responseHeader={status=0,QTime=1}},xxx.xx.xx.xxx:8983_solr={responseHeader={status=0,QTime=1}},xxx.xx.xx.xxx:8983_solr={responseHeader={status=0,QTime=1}},xxx.xx.xx.xxx:8983_solr={responseHeader={status=0,QTime=0}},xxx.xx.xx.xxx:8983_solr={responseHeader={status=0,QTime=0}},xxx.xx.xx.xxx:8983_solr={responseHeader={status=0,QTime=43}},xxx.xx.xx.xxx:8983_solr={responseHeader={status=0,QTime=131}},
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_replica1/={responseHeader={status=0,QTime=11871}}},1000455258538499159={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> 1000455258538499159 webapp=null path=/admin/cores
> params={async=1000455258538499159&qt=/admin/cores&collection.configName=COLLECTION&name=COLLECTION_shard2_0_replica1&action=CREATE&collection=COLLECTION&shard=shard2_0&wt=javabin&version=2}
> status=0
> QTime=1},1000455261568273501={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> 1000455261568273501 webapp=null path=/admin/cores
> params={async=1000455261568273501&qt=/admin/cores&collection.configName=COLLECTION&name=COLLECTION_shard2_1_replica1&action=CREATE&collection=COLLECTION&shard=shard2_1&wt=javabin&version=2}
> status=0
> QTime=1},1000455263597970918={responseHeader={status=0,QTime=1},STATUS=completed,Response=TaskId:
> 1000455263597970918 webapp=null path=/admin/cores
> params={nodeName=xxx.xx.xx.xxx:8983_solr&core=COLLECTION_shard2_1_replica1&async=1000455263597970918&qt=/admin/cores&coreNodeName=core_node6&action=PREPRECOVERY&checkLive=true&state=active&onlyIfLeader=true&wt=javabin&version=2}
> status=0
> QTime=1},1000455263617138064={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> 1000455263617138064 webapp=null path=/admin/cores
> params={core=COLLECTION_shard2_replica1&async=1000455263617138064&qt=/admin/cores&action=SPLIT&targetCore=COLLECTION_shard2_0_replica1&targetCore=COLLECTION_shard2_1_replica1&wt=javabin&version=2}
> status=0
> QTime=0},1000487066831162147={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> 1000487066831162147 webapp=null path=/admin/cores
> params={async=1000487066831162147&qt=/admin/cores&name=COLLECTION_shard2_1_replica1&action=REQUESTAPPLYUPDATES&wt=javabin&version=2}
> status=0 QTime=131}}
> INFO - 2017-08-23 21:58:38.156; [ ]
> org.apache.solr.common.cloud.SolrZkClient; makePath:
> /overseer/collection-map-running/mn-1001
> INFO - 2017-08-23 21:58:38.157; [ ]
> org.apache.solr.cloud.OverseerTaskProcessor; Overseer Collection Message
> Handler: Get the message id:/overseer/collection-queue-work/qn-0000000026
> message:{
> "collection":"COLLECTION",
> "shard":"shard3",
> "async":"1001",
> "operation":"splitshard"}
> INFO - 2017-08-23 21:58:38.202; [ ]
> org.apache.solr.cloud.OverseerCollectionMessageHandler;
> OverseerCollectionMessageHandler.processMessage : splitshard , {
> "collection":"COLLECTION",
> "shard":"shard3",
> "async":"1001",
> "operation":"splitshard"}
> INFO - 2017-08-23 21:58:38.246; [ ]
> org.apache.solr.cloud.OverseerCollectionMessageHandler; Split shard invoked
> ............
> ............
> INFO - 2017-08-24 07:41:18.387; [ ]
> org.apache.solr.cloud.overseer.SliceMutator; Update shard state shard3 to
> inactive
> INFO - 2017-08-24 07:41:18.387; [ ]
> org.apache.solr.cloud.overseer.SliceMutator; Update shard state shard3_0 to
> active
> INFO - 2017-08-24 07:41:18.387; [ ]
> org.apache.solr.cloud.overseer.SliceMutator; Update shard state shard3_1 to
> active
> INFO - 2017-08-24 07:41:18.391; [ ]
> org.apache.solr.cloud.overseer.ZkStateWriter; going to update_collection
> /collections/COLLECTION/state.json
> version: 2700
> INFO - 2017-08-24 07:41:18.395; [ ]
> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster state
> change: [WatchedEvent state:Syn
> cConnected type:NodeDataChanged path:/collections/COLLECTION/state.json]
> for collection [COLLECTION] has occurred - updating... (live nodes size: [8]
> )
> INFO - 2017-08-24 07:41:18.395; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for [COLLECTION]
> from [2700] to [2701]
> INFO - 2017-08-24 07:41:38.169; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1] org.apache.solr.update.DirectUpdateHandler2;
> start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> INFO - 2017-08-24 07:41:38.169; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.update.DirectUpdateHandler2; start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> INFO - 2017-08-24 07:41:38.253; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.update.DirectUpdateHandler2; start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> INFO - 2017-08-24 07:41:38.793; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrCore;
> SolrIndexSearcher has not changed - not re-opening:
> org.apache.solr.search.SolrIndexSearcher
> INFO - 2017-08-24 07:41:38.793; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrCore;
> SolrIndexSearcher has not changed - not re-opening:
> org.apache.solr.search.SolrIndexSearcher
> INFO - 2017-08-24 07:41:38.837; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
> INFO - 2017-08-24 07:41:38.976; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1] org.apache.solr.update.DirectUpdateHandler2;
> end_commit_flush
> INFO - 2017-08-24 07:41:38.976; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.update.processor.LogUpdateProc
> essorFactory$LogUpdateProcessor; [COLLECTION_shard2_0_replica1]
> webapp=/solr path=/update
> params={update.distrib=FROMLEADER&update.chain=add-unkn
>
> own-fields-to-the-schema&waitSearcher=true&openSearcher=true&commit=true&softCommit=true&distrib.from=
> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_sh
> ard3_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
> 0 807
> INFO - 2017-08-24 07:41:38.976; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1]
> org.apache.solr.update.processor.LogUpdateProcesso
> rFactory$LogUpdateProcessor; [COLLECTION_shard2_replica1] webapp=/solr
> path=/update params={update.distrib=FROMLEADER&update.chain=add-unknown-fi
>
> elds-to-the-schema&waitSearcher=true&openSearcher=true&commit=true&softCommit=true&distrib.from=
> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_shard3_r
> eplica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
> 0 857
> INFO - 2017-08-24 07:41:53.455; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.search.SolrIndexSearcher;
> Open
> ing Searcher@2b1e0336[COLLECTION_shard2_1_replica1] main
> INFO - 2017-08-24 07:41:53.503; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
> INFO - 2017-08-24 07:41:53.503; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.QuerySenderListener;
> QuerySenderListener sending requests to
> Searcher@2b1e0336[COLLECTION_shard2_1_replica1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.5.1):C1332771)
> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638)
> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033)
> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611)
> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083)
> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563)
> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123)
> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893)
> Uninverting(_s(5.5.1):c1969016) Uninverting(_v(5.5.1):c3842729)
> Uninverting(_w(5.5.1):c1660229) Uninverting(_x(5.5.1):c49355)
> Uninverting(_y(5.5.1):c150056) Uninverting(_z(5.5.1):c63319)
> Uninverting(_10(5.5.1):c80520) Uninverting(_11(5.5.1):c92203)
> Uninverting(_12(5.5.1):c96290) Uninverting(_13(5.5.1):c24621)
> Uninverting(_14(5.5.1):c94896) Uninverting(_15(5.5.1):c8987)
> Uninverting(_16(5.5.1):c12045) Uninverting(_17(5.5.1):c10943)
> Uninverting(_18(5.5.1):c7357) Uninverting(_19(5.5.1):c7519)
> Uninverting(_1a(5.5.1):c2426) Uninverting(_1b(5.5.1):c2700)
> Uninverting(_1c(5.5.1):c3336) Uninverting(_1d(5.5.1):c5069)
> Uninverting(_1e(5.5.1):c3707) Uninverting(_1f(5.5.1):c2391)
> Uninverting(_1g(5.5.1):c1389) Uninverting(_1h(5.5.1):c37)
> Uninverting(_1i(5.5.1):c42) Uninverting(_1j(5.5.1):c52)))}
> INFO - 2017-08-24 07:41:53.547; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.QuerySenderListener;
> QuerySenderListener done.
> INFO - 2017-08-24 07:41:53.547; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.SolrCore;
> [COLLECTION_shard2_1_replica1] Registered new searcher
> Searcher@2b1e0336[COLLECTION_shard2_1_replica1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.5.1):C1332771)
> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638)
> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033)
> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611)
> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083)
> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563)
> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123)
> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893)
> Uninverting(_s(5.5.1):c1969016) Uninverting(_v(5.5.1):c3842729)
> Uninverting(_w(5.5.1):c1660229) Uninverting(_x(5.5.1):c49355)
> Uninverting(_y(5.5.1):c150056) Uninverting(_z(5.5.1):c63319)
> Uninverting(_10(5.5.1):c80520) Uninverting(_11(5.5.1):c92203)
> Uninverting(_12(5.5.1):c96290) Uninverting(_13(5.5.1):c24621)
> Uninverting(_14(5.5.1):c94896) Uninverting(_15(5.5.1):c8987)
> Uninverting(_16(5.5.1):c12045) Uninverting(_17(5.5.1):c10943)
> Uninverting(_18(5.5.1):c7357) Uninverting(_19(5.5.1):c7519)
> Uninverting(_1a(5.5.1):c2426) Uninverting(_1b(5.5.1):c2700)
> Uninverting(_1c(5.5.1):c3336) Uninverting(_1d(5.5.1):c5069)
> Uninverting(_1e(5.5.1):c3707) Uninverting(_1f(5.5.1):c2391)
> Uninverting(_1g(5.5.1):c1389) Uninverting(_1h(5.5.1):c37)
> Uninverting(_1i(5.5.1):c42) Uninverting(_1j(5.5.1):c52)))}
> INFO - 2017-08-24 07:41:56.237; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [COLLECTION_shard2_1_replica1] webapp=/solr path=/update
> params={update.distrib=FROMLEADER&update.chain=add-unknown-fields-to-the-schema&waitSearcher=true&openSearcher=true&commit=true&softCommit=true&distrib.from=
> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_shard3_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
> 0 18117
> INFO - 2017-08-24 07:42:24.803; [ ]
> org.apache.solr.common.cloud.SolrZkClient; makePath:
> /overseer/collection-map-completed/mn-1001
> INFO - 2017-08-24 07:42:24.856; [ ]
> org.apache.solr.cloud.OverseerTaskProcessor$Runner; Overseer Collection
> Message Handler: Message id:/overseer/collection-queue-work/qn-0000000026
> complete,
> response:{success={xxx.xx.xx.xxx:8984_solr={responseHeader={status=0,QTime=4}},xxx.xx.xx.xxx:8984_solr={responseHeader={status=0,QTime=0}},xxx.xx.xx.xxx:8984_solr={responseHeader={status=0,QTime=1}},xxx.xx.xx.xxx:8984_solr={responseHeader={status=0,QTime=1}},xxx.xx.xx.xxx:8984_solr={responseHeader={status=0,QTime=1}},xxx.xx.xx.xxx:8984_solr={responseHeader={status=0,QTime=0}},xxx.xx.xx.xxx:8984_solr={responseHeader={status=0,QTime=189}},
> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_shard3_replica1/={responseHeader={status=0,QTime=66236}}},1001487080480985685={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> 1001487080480985685 webapp=null path=/admin/cores
> params={async=1001487080480985685&qt=/admin/cores&collection.configName=COLLECTION&name=COLLECTION_shard3_0_replica1&action=CREATE&collection=COLLECTION&shard=shard3_0&wt=javabin&version=2}
> status=0
> QTime=4},1001487084675100247={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> 1001487084675100247 webapp=null path=/admin/cores
> params={async=1001487084675100247&qt=/admin/cores&collection.configName=COLLECTION&name=COLLECTION_shard3_1_replica1&action=CREATE&collection=COLLECTION&shard=shard3_1&wt=javabin&version=2}
> status=0
> QTime=0},1001487086698326060={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> 1001487086698326060 webapp=null path=/admin/cores
> params={nodeName=xxx.xx.xx.xxx:8984_solr&core=COLLECTION_shard3_1_replica1&async=1001487086698326060&qt=/admin/cores&coreNodeName=core_node8&action=PREPRECOVERY&checkLive=true&state=active&onlyIfLeader=true&wt=javabin&version=2}
> status=0
> QTime=1},1001487086756758564={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> 1001487086756758564 webapp=null path=/admin/cores
> params={core=COLLECTION_shard3_replica1&async=1001487086756758564&qt=/admin/cores&action=SPLIT&targetCore=COLLECTION_shard3_0_replica1&targetCore=COLLECTION_shard3_1_replica1&wt=javabin&version=2}
> status=0
> QTime=1},1001522038777271446={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> 1001522038777271446 webapp=null path=/admin/cores
> params={async=1001522038777271446&qt=/admin/cores&name=COLLECTION_shard3_1_replica1&action=REQUESTAPPLYUPDATES&wt=javabin&version=2}
> status=0 QTime=189}}
*Log messages during next restart:*
> INFO - 2017-08-24 08:52:37.821; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.handler.ReplicationHandler;
> Co
> mmits will be reserved for 10000
> INFO - 2017-08-24 08:52:37.826; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrDeletionPolicy;
> SolrD
> eletionPolicy.onInit: commits: num=1
>
> commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solrcloud/solr-5.5.1/example/cloud/node1/solr/COLLECTION_shard2_0_replica1/data/index
> lo
> ckFactory=org.apache.lucene.store.NativeFSLockFactory@725fe252;
> maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
> INFO - 2017-08-24 08:52:37.826; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrDeletionPolicy;
> newes
> t commit generation = 1
> INFO - 2017-08-24 08:52:37.827; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.search.SolrIndexSearcher;
> Open
> ing Searcher@5c49571c[COLLECTION_shard2_0_replica1] main
> INFO - 2017-08-24 08:52:37.827; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 1, message = {
> "core":"COLLECTION-digital-archive_shard1_replica1",
> "core_node_name":"core_node1",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8983/solr";,
> "node_name":"xxx.xx.xx.xxx:8983_solr",
> "numShards":"1",
> "state":"active",
> "shard":"shard1",
> "collection":"COLLECTION-digital-archive",
> "operation":"state"} current state version: 0
> INFO - 2017-08-24 08:52:37.827; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud.ZkStateReader;
> Load collection config from: [/collections/COLLECTION]
> INFO - 2017-08-24 08:52:37.827; [ ]
> org.apache.solr.cloud.overseer.ReplicaMutator; Update state numShards=1
> message={
> "core":"COLLECTION-digital-archive_shard1_replica1",
> "core_node_name":"core_node1",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8983/solr";,
> "node_name":"xxx.xx.xx.xxx:8983_solr",
> "numShards":"1",
> "state":"active",
> "shard":"shard1",
> "collection":"COLLECTION-digital-archive",
> "operation":"state"}
> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud.ZkStateReader;
> path=[/collections/COLLECTION] [configName]=[COLLECTION] specified config
> exists in ZooKeeper
> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage; Setting up ZooKeeper-based
> storage for the RestManager with znodeBase: /configs/COLLECTION
> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrDeletionPolicy;
> SolrDeletionPolicy.onInit: commits: num=1
> commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solrdata/index
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@725fe252;
> maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_33sj,generation=144883}
> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.UpdateLog; Looking
> up max value of version field to seed version buckets
> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2 r:core_node1
> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrDeletionPolicy;
> newest commit generation = 144883
> INFO - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.QuerySenderListener;
> QuerySenderListener sending requests to
> Searcher@269e2d59[COLLECTION_shard2_1_replica1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.QuerySenderListener;
> QuerySenderListener done.
> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener;
> Loading spell index for spellchecker: default
> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.VersionInfo;
> Refreshing highest value of _version_ for 65536 version buckets from index
> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.SolrCore;
> [COLLECTION_shard2_1_replica1] Registered new searcher
> Searcher@269e2d59[COLLECTION_shard2_1_replica1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.VersionInfo; No
> terms found for _version_, cannot seed version bucket highest value from
> index
> INFO - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; Configured
> ZooKeeperStorageIO with znodeBase: /configs/COLLECTION
> INFO - 2017-08-24 08:52:37.830; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.RestManager;
> Initializing RestManager with initArgs: {}
> INFO - 2017-08-24 08:52:37.830; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.UpdateLog; Could not
> find max version in index or recent updates, using new clock
> 1576601919506350080
> INFO - 2017-08-24 08:52:37.830; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage; Reading _rest_managed.json
> using ZooKeeperStorageIO:path=/configs/COLLECTION
> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; Read 33
> bytes from znode /configs/COLLECTION/_rest_managed.json
> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage; Loaded LinkedHashMap at path
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/COLLECTION
> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource;
> Loaded initArgs {} for /rest/managed
> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.RestManager;
> Initializing 2 registered ManagedResources
> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage; Reading
> _schema_analysis_stopwords_english.json using
> ZooKeeperStorageIO:path=/configs/COLLECTION
> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.UpdateLog; Took
> 3.0ms to seed version buckets with highest version 1576601919506350080
> INFO - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; watch
> zkdir /configs/COLLECTION
> INFO - 2017-08-24 08:52:37.833; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.CoreContainer;
> registering core: COLLECTION_shard2_1_replica1
> INFO - 2017-08-24 08:52:37.833; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; Read 450
> bytes from znode /configs/COLLECTION/_schema_analysis_stopwords_english.json
> INFO - 2017-08-24 08:52:37.834; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController;
> Register replica - core:COLLECTION_shard2_1_replica1 address:
> http://xxx.xx.xx.xxx:8983/solr collection:COLLECTION shard:shard2_1
> INFO - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage; Loaded LinkedHashMap at path
> _schema_analysis_stopwords_english.json using
> ZooKeeperStorageIO:path=/configs/COLLECTION
> INFO - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource;
> Loaded initArgs {ignoreCase=true} for /schema/analysis/stopwords/english
> INFO - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.schema.analysis.ManagedWordSetResource; Loaded 35
> words for /schema/analysis/stopwords/english
> INFO - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource;
> Notified 1 observers of /schema/analysis/stopwords/english
> INFO - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage; Reading
> _schema_analysis_synonyms_english.json using
> ZooKeeperStorageIO:path=/configs/COLLECTION
> INFO - 2017-08-24 08:52:37.837; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.LeaderElector; Joined
> leadership election with path:
> /collections/COLLECTION/leader_elect/shard2_1/election/242652799835373569-core_node6-n_0000000003
> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; Read 172
> bytes from znode /configs/COLLECTION/_schema_analysis_synonyms_english.json
> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.ManagedResourceStorage; Loaded LinkedHashMap at path
> _schema_analysis_synonyms_english.json using
> ZooKeeperStorageIO:path=/configs/COLLECTION
> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader
> process for shard=shard2_1 and weAreReplacement=false and
> leaderVoteWait=180000
> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource;
> Loaded initArgs {ignoreCase=true,format=solr} for
> /schema/analysis/synonyms/english
> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager;
> Loaded 3 synonym mappings for /schema/analysis/synonyms/english
> INFO - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource;
> Notified 1 observers of /schema/analysis/synonyms/english
> INFO - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.handler.component.SpellCheckComponent; Initializing spell
> checkers
> INFO - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.spelling.DirectSolrSpellChecker; init:
> {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
> INFO - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.handler.component.SpellCheckComponent; No queryConverter
> defined, using default converter
> INFO - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud.ZkStateReader;
> Load collection config from: [/collections/COLLECTION]
> INFO - 2017-08-24 08:52:37.840; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud.ZkStateReader;
> path=[/collections/COLLECTION] [configName]=[COLLECTION] specified config
> exists in ZooKeeper
> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas found to
> continue.
> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new leader -
> try and sync
> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.SyncStrategy; Sync
> replicas to http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/
> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.SyncStrategy; Sync
> Success - now sync replicas to me
> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.handler.ReplicationHandler;
> Commits will be reserved for 10000
> INFO - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.SyncStrategy;
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ has no
> replicas
> INFO - 2017-08-24 08:52:37.844; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 1, message = {
> "operation":"leader",
> "shard":"shard2_1",
> "collection":"COLLECTION"} current state version: 0
> INFO - 2017-08-24 08:52:37.844; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.cloud.ShardLeaderElectionContextBase$1; Creating leader
> registration node /collections/COLLECTION/leaders/shard2_1/leader after
> winning as
> /collections/COLLECTION/leader_elect/shard2_1/election/242652799835373569-core_node6-n_0000000003
> INFO - 2017-08-24 08:52:37.847; [ ]
> org.apache.solr.cloud.overseer.ZkStateWriter; going to update_collection
> /collections/COLLECTION-digital-archive/state.json version: 902
> INFO - 2017-08-24 08:52:37.850; [ ]
> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/COLLECTION-digital-archive/state.json] for collection
> [COLLECTION-digital-archive] has occurred - updating... (live nodes size:
> [1])
> INFO - 2017-08-24 08:52:37.852; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for
> [COLLECTION-digital-archive] from [902] to [903]
> INFO - 2017-08-24 08:52:37.852; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1]
> org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader:
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ shard2_1
> INFO - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.QuerySenderListener;
> QuerySenderListener sending requests to
> Searcher@5c49571c[COLLECTION_shard2_0_replica1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> INFO - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; We are
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ and leader
> is http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/
> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog; Looking
> up max value of version field to seed version buckets
> INFO - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.QuerySenderListener;
> QuerySenderListener done.
> INFO - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; We are
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ and leader
> is http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/
> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog; Looking
> up max value of version field to seed version buckets
> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener;
> Loading spell index for spellchecker: default
> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.VersionInfo;
> Refreshing highest value of _version_ for 65536 version buckets from index
> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.VersionInfo; No
> terms found for _version_, cannot seed version bucket highest value from
> index
> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; No
> LogReplay needed for core=COLLECTION_shard2_1_replica1 baseURL=
> http://xxx.xx.xx.xxx:8983/solr
> INFO - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; I am
> the leader, no recovery necessary
> INFO - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog; Could not
> find max version in index or recent updates, using new clock
> 1576601919533613056
> INFO - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrCore;
> [COLLECTION_shard2_0_replica1] Registered new searcher
> Searcher@5c49571c[COLLECTION_shard2_0_replica1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> INFO - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController;
> publishing state=active
> INFO - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_1 r:core_node6
> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController;
> numShards not found on descriptor - reading it from system property
> INFO - 2017-08-24 08:52:37.857; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog; Took
> 2.0ms to seed version buckets with highest version 1576601919533613056
> INFO - 2017-08-24 08:52:37.857; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.CoreContainer;
> registering core: COLLECTION_shard2_0_replica1
> INFO - 2017-08-24 08:52:37.858; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController;
> Register replica - core:COLLECTION_shard2_0_replica1 address:
> http://xxx.xx.xx.xxx:8983/solr collection:COLLECTION shard:shard2_0
> INFO - 2017-08-24 08:52:37.862; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.LeaderElector; Joined
> leadership election with path:
> /collections/COLLECTION/leader_elect/shard2_0/election/242652799835373569-core_node5-n_0000000003
> INFO - 2017-08-24 08:52:37.864; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader
> process for shard=shard2_0 and weAreReplacement=false and
> leaderVoteWait=180000
> INFO - 2017-08-24 08:52:37.867; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 2, message = {
> "operation":"leader",
> "shard":"shard2_1",
> "collection":"COLLECTION",
> "base_url":"http://xxx.xx.xx.xxx:8983/solr";,
> "core":"COLLECTION_shard2_1_replica1",
> "state":"active"} current state version: 0
> INFO - 2017-08-24 08:52:37.871; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas found to
> continue.
> INFO - 2017-08-24 08:52:37.871; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new leader -
> try and sync
> INFO - 2017-08-24 08:52:37.872; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.SyncStrategy; Sync
> replicas to http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/
> INFO - 2017-08-24 08:52:37.872; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.SyncStrategy; Sync
> Success - now sync replicas to me
> INFO - 2017-08-24 08:52:37.872; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.SyncStrategy;
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/ has no
> replicas
> INFO - 2017-08-24 08:52:37.874; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.cloud.ShardLeaderElectionContextBase$1; Creating leader
> registration node /collections/COLLECTION/leaders/shard2_0/leader after
> winning as
> /collections/COLLECTION/leader_elect/shard2_0/election/242652799835373569-core_node5-n_0000000003
> INFO - 2017-08-24 08:52:37.874; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 2, message = {
> "core":"COLLECTION_shard2_1_replica1",
> "core_node_name":"core_node6",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8983/solr";,
> "node_name":"xxx.xx.xx.xxx:8983_solr",
> "state":"active",
> "shard":"shard2_1",
> "collection":"COLLECTION",
> "operation":"state"} current state version: 0
> INFO - 2017-08-24 08:52:37.876; [ ]
> org.apache.solr.cloud.overseer.ReplicaMutator; Update state numShards=null
> message={
> "core":"COLLECTION_shard2_1_replica1",
> "core_node_name":"core_node6",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8983/solr";,
> "node_name":"xxx.xx.xx.xxx:8983_solr",
> "state":"active",
> "shard":"shard2_1",
> "collection":"COLLECTION",
> "operation":"state"}
> INFO - 2017-08-24 08:52:37.881; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1]
> org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader:
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/ shard2_0
> INFO - 2017-08-24 08:52:37.882; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 2, message = {
> "operation":"leader",
> "shard":"shard2_0",
> "collection":"COLLECTION"} current state version: 0
> INFO - 2017-08-24 08:52:37.882; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; We are
> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/ and leader
> is http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/
> INFO - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; No
> LogReplay needed for core=COLLECTION_shard2_0_replica1 baseURL=
> http://xxx.xx.xx.xxx:8983/solr
> INFO - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; I am
> the leader, no recovery necessary
> INFO - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController;
> publishing state=active
> INFO - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5
> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController;
> numShards not found on descriptor - reading it from system property
> INFO - 2017-08-24 08:52:37.888; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 2, message = {
> "operation":"leader",
> "shard":"shard2_0",
> "collection":"COLLECTION",
> "base_url":"http://xxx.xx.xx.xxx:8983/solr";,
> "core":"COLLECTION_shard2_0_replica1",
> "state":"active"} current state version: 0
> INFO - 2017-08-24 08:52:37.894; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 1, message = {
> "core":"COLLECTION_shard2_0_replica1",
> "core_node_name":"core_node5",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8983/solr";,
> "node_name":"xxx.xx.xx.xxx:8983_solr",
> "state":"active",
> "shard":"shard2_0",
> "collection":"COLLECTION",
> "operation":"state"} current state version: 0
> INFO - 2017-08-24 08:52:37.894; [ ]
> org.apache.solr.cloud.overseer.ReplicaMutator; Update state numShards=null
> message={
> "core":"COLLECTION_shard2_0_replica1",
> "core_node_name":"core_node5",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8983/solr";,
> "node_name":"xxx.xx.xx.xxx:8983_solr",
> "state":"active",
> "shard":"shard2_0",
> "collection":"COLLECTION",
> "operation":"state"}
> INFO - 2017-08-24 08:52:38.000; [ ]
> org.apache.solr.cloud.overseer.ZkStateWriter; going to update_collection
> /collections/COLLECTION/state.json version: 2721
> INFO - 2017-08-24 08:52:38.005; [ ]
> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/COLLECTION/state.json] for collection [COLLECTION] has
> occurred - updating... (live nodes size: [1])
> INFO - 2017-08-24 08:52:38.006; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for [COLLECTION]
> from [2721] to [2722]
> INFO - 2017-08-24 08:52:40.805; [ ]
> org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher; A live node
> change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged
> path:/live_nodes], has occurred - updating... (live nodes size: [1])
> INFO - 2017-08-24 08:52:40.931; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 1, message = {
> "core":"COLLECTION1_shard4_replica1",
> "core_node_name":"core_node3",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8984/solr";,
> "node_name":"xxx.xx.xx.xxx:8984_solr",
> "numShards":"4",
> "state":"down",
> "shard":"shard4",
> "collection":"COLLECTION1",
> "operation":"state"} current state version: 0
> INFO - 2017-08-24 08:52:40.931; [ ]
> org.apache.solr.cloud.overseer.ReplicaMutator; Update state numShards=4
> message={
> "core":"COLLECTION1_shard4_replica1",
> "core_node_name":"core_node3",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8984/solr";,
> "node_name":"xxx.xx.xx.xxx:8984_solr",
> "numShards":"4",
> "state":"down",
> "shard":"shard4",
> "collection":"COLLECTION1",
> "operation":"state"}
> INFO - 2017-08-24 08:52:40.934; [ ]
> org.apache.solr.cloud.overseer.ZkStateWriter; going to update_collection
> /collections/COLLECTION1/state.json version: 429
> INFO - 2017-08-24 08:52:40.935; [ ]
> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/COLLECTION1/state.json] for collection [COLLECTION1] has
> occurred - updating... (live nodes size: [2])
> INFO - 2017-08-24 08:52:40.936; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for [COLLECTION1]
> from [429] to [430]
> INFO - 2017-08-24 08:52:40.940; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 4, message = {
> "core":"COLLECTION_shard3_0_replica1",
> "core_node_name":"core_node7",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8984/solr";,
> "node_name":"xxx.xx.xx.xxx:8984_solr",
> "state":"down",
> "shard":"shard3_0",
> "collection":"COLLECTION",
> "operation":"state"} current state version: 0
> INFO - 2017-08-24 08:52:40.940; [ ]
> org.apache.solr.cloud.overseer.ReplicaMutator; Update state numShards=null
> message={
> "core":"COLLECTION_shard3_0_replica1",
> "core_node_name":"core_node7",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8984/solr";,
> "node_name":"xxx.xx.xx.xxx:8984_solr",
> "state":"down",
> "shard":"shard3_0",
> "collection":"COLLECTION",
> "operation":"state"}
> INFO - 2017-08-24 08:52:40.946; [ ]
> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> queueSize: 0, message = {
> "core":"COLLECTION_sandbox_shard4_replica1",
> "core_node_name":"core_node4",
> "roles":null,
> "base_url":"http://xxx.xx.xx.xxx:8984/solr";,
> "node_name":"xxx.xx.xx.xxx:8984_solr",
> "numShards":"4",
> "state":"down",
> "shard":"shard4",
Thanks,
Vanniarajan
On Thu, Aug 24, 2017 at 8:36 PM, Binoy Dalal <binoydala...@gmail.com> wrote:
> Vanniarajan,
> Do you see errors in your solr logs when you reboot? If so paste them here.
> Are both the shards up?
> Have you checked the data directory for both and confirmed that the data is
> gone?
>
> On Thu 24 Aug, 2017, 18:04 Vannia Rajan <van...@knackforge.com> wrote:
>
> > Hi,
> >
> > I'm facing weird issues on using SPLITSHARD on fairly large shards (150GB
> > shard size), using SOLR 5.5.1 in cloud mode (4 nodes, 1 shard per node).
> >
> > After issuing SPLITSHARD, I successfully get the sub-shards activated and
> > my parent shard set "inactive". The newly split shards counts equal the
> > parent's counts and all works well. The issue is when I restart SOLR, the
> > newly split shards gets reduced to 71KB and 0 documents.
> >
> > Has anyone faced such issues, and do you think this can be a possible bug
> > that could be fixed if I upgrade SOLR to 6.6?
> >
> > Also, I would like to know if there is a way to "rollback" the
> SPLITSHARD.
> > I would like to reactivate my parent shard with all documents intact and
> > delete the new sub-shards. I have no idea how I could reactivate a shard
> > that is now marked as "inactive".
> >
> > Thanks,
> > Vanniarajan
> >
> --
> Regards,
> Binoy Dalal
>
