Thanks for sharing your workaround!
Agree that error message is extremely unhelpful :)
Would you care to report it as a bug in JIRA?
Jan
> 15. apr. 2020 kl. 21:54 skrev Eugene Livis <eli...@basistech.com>:
>
> In case this helps somebody in the future, given how completely unhelpful
> the Solr error message is - turns out the problem was occurring because in
> solrconfig.xml the updateLog was disabled. I have enabled updateLog the
> following way and "restore" operation started working:
>
> <updateLog>
> <str name="dir">${solr.ulog.dir:}</str>
> <int
> name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
> </updateLog>
>
>
> On Wed, Apr 8, 2020 at 5:17 PM Eugene Livis <eli...@basistech.com> wrote:
>
>> Hello,
>>
>> I have been unsuccessfully trying to find a way to restore a collection
>> using the Collections API for the last several days. *I would
>> greatly appreciate any help as I am now stuck.* I am using Solr 8.2 in
>> cloud mode. To simplify things, I have only a single Solr node on my local
>> machine (though I have also tried multi-node setups). I am able to
>> successfully create collections, index documents, and run searches against
>> the index. General gist of what I am trying to do is to be able to backup
>> an existing collection, delete that collection, and then restore the
>> collection from backup when needed. Our use case requires creating
>> thousands of collections, which I believe is highly not recommended in Solr
>> Cloud (is that correct?), so this backup-delete-restore mechanism is our
>> way of reducing the number of collections in Solr at any given time. Just
>> FYI, in our use case it is ok for searches to take a couple of minutes to
>> produce results.
>>
>> So far I have not been able to restore a single collection using the
>> collection RESTORE API. I am able to create backups by running the
>> following HTTP command in Solr admin console:
>>
>>
>> http://localhost:8983/solr/admin/collections?action=BACKUP&name=test1_backup2&collection=test1_20200407_170438_20200407_170441&location=C:\TEST\DELETE\BACKUPS
>>
>>
>> The backup appears to be successfully created. It contains a snapshot from
>> "shard1", which is the only shard in the cluster. It also contains the
>> "zk_backup" directory, which contains my Solr config and
>> "collection_state.json" file. And it contains a "backup.properties" file.
>> Everything looks good to me.
>>
>> However, when I attempt to restore the collection using the following HTTP
>> command:
>>
>>
>> http://localhost:8983/solr/admin/collections?action=RESTORE&name=test1_backup2&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW
>>
>>
>> I get the same extremely vague error messages in Solr logs:
>>
>>
>> *RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing
>> SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create core
>> [test1_backup_NEW_shard1_replica_n1] Caused by: nullERROR
>> (qtp1099855928-24) [c:test1_backup_NEW ] o.a.s.s.HttpSolrCall
>> null:org.apache.solr.common.SolrException: ADDREPLICA failed to create
>> replica *
>>
>> Below is the full Solr log of the restore command:
>>
>> 2020-04-08 21:10:56.753 INFO (qtp1099855928-24) [ ]
>> o.a.s.h.a.CollectionsHandler Invoked Collection Action :restore with params
>> name=test1_backup2&action=RESTORE&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW
>> and sendToOCPQueue=true
>> 2020-04-08 21:10:56.783 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Using existing config
>> AutopsyConfig
>> 2020-04-08 21:10:56.783 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Starting restore into
>> collection=test1_backup_NEW with backup_name=test1_backup2 at
>> location=file:///C:/TEST/DELETE/BACKUPS/
>> 2020-04-08 21:10:56.784 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.CreateCollectionCmd Create collection
>> test1_backup_NEW
>> 2020-04-08 21:10:56.900 WARN
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to
>> create a collection (test1_backup_NEW) without cores.
>> 2020-04-08 21:10:56.908 INFO
>> (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
>> ] o.a.s.c.o.SliceMutator Update shard state invoked for collection:
>> test1_backup_NEW with message: {
>> "shard1":"construction",
>> "collection":"test1_backup_NEW",
>> "operation":"updateshardstate"}
>> 2020-04-08 21:10:56.908 INFO
>> (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
>> ] o.a.s.c.o.SliceMutator Update shard state shard1 to construction
>> 2020-04-08 21:10:56.925 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Adding replica for
>> shard=shard1
>> collection=DocCollection(test1_backup_NEW//collections/test1_backup_NEW/state.json/0)={
>> "pullReplicas":0,
>> "replicationFactor":1,
>> "shards":{"shard1":{
>> "range":"80000000-7fffffff",
>> "state":"active",
>> "replicas":{}}},
>> "router":{"name":"compositeId"},
>> "maxShardsPerNode":"1",
>> "autoAddReplicas":"false",
>> "nrtReplicas":1,
>> "tlogReplicas":0}
>> 2020-04-08 21:10:56.928 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.AddReplicaCmd Node Identified
>> localhost:8983_solr for creating new replica of shard shard1 for collection
>> test1_backup_NEW
>> 2020-04-08 21:10:56.939 INFO
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica
>> command.
>> 2020-04-08 21:10:56.945 INFO
>> (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [
>> ] o.a.s.c.o.SliceMutator createReplica() {
>> "operation":"addreplica",
>> "collection":"test1_backup_NEW",
>> "shard":"shard1",
>> "core":"test1_backup_NEW_shard1_replica_n1",
>> "state":"down",
>> "base_url":"http://localhost:8983/solr";,
>> "node_name":"localhost:8983_solr",
>> "type":"NRT"}
>> 2020-04-08 21:10:57.348 INFO (qtp1099855928-19) [
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core
>> create command
>> qt=/admin/cores&coreNodeName=core_node2&collection.configName=AutopsyConfig&name=test1_backup_NEW_shard1_replica_n1&action=CREATE&collection=test1_backup_NEW&shard=shard1&wt=javabin&version=2&replicaType=NRT
>> 2020-04-08 21:10:57.373 WARN (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.SolrConfig Couldn't add files from C:\total\crap\dir\ignored
>> filtered by null to classpath: C:\total\crap\dir\ignored
>> 2020-04-08 21:10:57.374 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.SolrResourceLoader [test1_backup_NEW_shard1_replica_n1] Added 56
>> libs to classloader, from paths:
>> [/C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/clustering/lib,
>> /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/extraction/lib,
>> /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/langid/lib,
>> /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/velocity/lib]
>> 2020-04-08 21:10:57.395 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.2.0
>> 2020-04-08 21:10:57.424 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.s.IndexSchema [test1_backup_NEW_shard1_replica_n1] Schema
>> name=Autopsy Keyword Search
>> 2020-04-08 21:10:57.451 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.s.IndexSchema Loaded schema Autopsy Keyword Search/2.2 with uniqueid
>> field id
>> 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.CoreContainer Creating SolrCore
>> 'test1_backup_NEW_shard1_replica_n1' using configuration from collection
>> test1_backup_NEW, trusted=true
>> 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.m.r.SolrJmxReporter JMX monitoring for
>> 'solr.core.test1_backup_NEW.shard1.replica_n1' (registry
>> 'solr.core.test1_backup_NEW.shard1.replica_n1') enabled at server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@214b199c
>> 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.SolrCore [[test1_backup_NEW_shard1_replica_n1] ] Opening new
>> SolrCore at
>> [C:\Bitnami\solr-8.2.0-2\apache-solr\server\solr\test1_backup_NEW_shard1_replica_n1],
>> dataDir=[C:\Bitnami\solr-8.2.0-2\apache-solr\server\solr\test1_backup_NEW_shard1_replica_n1\data\]
>> 2020-04-08 21:10:57.491 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
>> 2020-04-08 21:10:57.527 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.u.CommitTracker Hard AutoCommit: if 15000 uncommitted docs;
>> 2020-04-08 21:10:57.527 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.u.CommitTracker Soft AutoCommit: disabled
>> 2020-04-08 21:10:57.535 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.s.SolrIndexSearcher Opening
>> [Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
>> main]
>> 2020-04-08 21:10:57.536 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
>> znodeBase: /configs/AutopsyConfig
>> 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
>> ZooKeeperStorageIO:path=/configs/AutopsyConfig
>> 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.h.c.SpellCheckComponent Initializing spell checkers
>> 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.s.DirectSolrSpellChecker init:
>> {name=default,field=name,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
>> 2020-04-08 21:10:57.540 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
>> 2020-04-08 21:10:57.541 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.c.SolrCore [test1_backup_NEW_shard1_replica_n1] CLOSING SolrCore
>> org.apache.solr.core.SolrCore@3a4eb909
>> 2020-04-08 21:10:57.541 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.QuerySenderListener
>> QuerySenderListener sending requests to
>> Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>> 2020-04-08 21:10:57.541 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.m.SolrMetricManager Closing metric reporters for
>> registry=solr.core.test1_backup_NEW.shard1.replica_n1, tag=3a4eb909
>> 2020-04-08 21:10:57.542 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.m.r.SolrJmxReporter Closing reporter
>> [org.apache.solr.metrics.reporters.SolrJmxReporter@54eb018f: rootName =
>> null, domain = solr.core.test1_backup_NEW.shard1.replica_n1, service url =
>> null, agent id = null] for registry
>> solr.core.test1_backup_NEW.shard1.replica_n1 /
>> com.codahale.metrics.MetricRegistry@55bc77fd
>> 2020-04-08 21:10:57.542 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.S.Request
>> [test1_backup_NEW_shard1_replica_n1] webapp=null path=null
>> params={q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false&event=firstSearcher}
>> hits=0 status=0 QTime=1
>> 2020-04-08 21:10:57.542 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.QuerySenderListener
>> QuerySenderListener done.
>> 2020-04-08 21:10:57.542 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading
>> spell index for spellchecker: default
>> 2020-04-08 21:10:57.542 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading
>> spell index for spellchecker: wordbreak
>> 2020-04-08 21:10:57.542 INFO
>> (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr
>> x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1
>> r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2
>> x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.SolrCore
>> [test1_backup_NEW_shard1_replica_n1] Registered new searcher
>> Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1]
>> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>> 2020-04-08 21:10:57.552 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.m.SolrMetricManager Closing metric reporters for
>> registry=solr.collection.test1_backup_NEW.shard1.leader, tag=3a4eb909
>> 2020-04-08 21:10:57.560 ERROR (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error
>> CREATEing SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create
>> core [test1_backup_NEW_shard1_replica_n1] Caused by: null
>> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1195)
>> at
>> org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
>> at
>> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
>> at
>> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:396)
>> at
>> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)
>> at
>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at
>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at org.eclipse.jetty.server.Server.handle(Server.java:505)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
>> at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:311)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125)
>> at
>> org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348)
>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>> at
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.solr.common.SolrException: Unable to create core
>> [test1_backup_NEW_shard1_replica_n1]
>> at
>> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1266)
>> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1156)
>> ... 50 more
>> Caused by: org.apache.solr.common.SolrException
>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1055)
>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:882)
>> at
>> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1245)
>> ... 51 more
>> Caused by: java.lang.NullPointerException
>> at
>> org.apache.solr.core.SolrCore.bufferUpdatesIfConstructing(SolrCore.java:1097)
>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1029)
>> ... 53 more
>>
>> 2020-04-08 21:10:57.561 INFO (qtp1099855928-19) [c:test1_backup_NEW
>> s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
>> params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=AutopsyConfig&name=test1_backup_NEW_shard1_replica_n1&action=CREATE&collection=test1_backup_NEW&shard=shard1&wt=javabin&version=2&replicaType=NRT}
>> status=400 QTime=213
>> 2020-04-08 21:10:57.577 ERROR
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.OverseerCollectionMessageHandler Error
>> from shard: http://localhost:8983/solr =>
>> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
>> Error from server at null: Error CREATEing SolrCore
>> 'test1_backup_NEW_shard1_replica_n1': Unable to create core
>> [test1_backup_NEW_shard1_replica_n1] Caused by: null
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
>> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
>> Error from server at null: Error CREATEing SolrCore
>> 'test1_backup_NEW_shard1_replica_n1': Unable to create core
>> [test1_backup_NEW_shard1_replica_n1] Caused by: null
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
>> ~[?:?]
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400)
>> ~[?:?]
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
>> ~[?:?]
>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274)
>> ~[?:?]
>> at
>> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
>> ~[?:?]
>> at
>> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
>> ~[?:?]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[?:1.8.0_222-1-ojdkbuild]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[?:1.8.0_222-1-ojdkbuild]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[?:1.8.0_222-1-ojdkbuild]
>> at
>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
>> ~[metrics-core-4.0.5.jar:4.0.5]
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
>> ~[?:?]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> ~[?:1.8.0_222-1-ojdkbuild]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> ~[?:1.8.0_222-1-ojdkbuild]
>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222-1-ojdkbuild]
>> 2020-04-08 21:10:57.578 ERROR
>> (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr)
>> [c:test1_backup_NEW ] o.a.s.c.a.c.OverseerCollectionMessageHandler
>> Collection: test1_backup_NEW operation: restore
>> failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create
>> replica
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1030)
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1013)
>> at
>> org.apache.solr.cloud.api.collections.AddReplicaCmd.lambda$addReplica$1(AddReplicaCmd.java:177)
>> at
>> org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:199)
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.addReplica(OverseerCollectionMessageHandler.java:708)
>> at
>> org.apache.solr.cloud.api.collections.RestoreCmd.call(RestoreCmd.java:286)
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
>> at
>> org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by:
>> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:
>> Error from server at null: Error CREATEing SolrCore
>> 'test1_backup_NEW_shard1_replica_n1': Unable to create core
>> [test1_backup_NEW_shard1_replica_n1] Caused by: null
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727)
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400)
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274)
>> at
>> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
>> at
>> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
>> ... 4 more
>>
>> 2020-04-08 21:10:57.604 INFO (qtp1099855928-24) [c:test1_backup_NEW ]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
>> params={name=test1_backup2&action=RESTORE&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW}
>> status=500 QTime=850
>> 2020-04-08 21:10:57.605 ERROR (qtp1099855928-24) [c:test1_backup_NEW ]
>> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: ADDREPLICA
>> failed to create replica
>> at
>> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)
>> at
>> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)
>> at
>> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:252)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)
>> at
>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at
>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> at org.eclipse.jetty.server.Server.handle(Server.java:505)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
>> at
>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>> at
>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>> at
>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> 2020-04-08 21:10:58.775 INFO
>> (OverseerCollectionConfigSetProcessor-72057883336638464-localhost:8983_solr-n_0000000003)
>> [ ] o.a.s.c.OverseerTaskQueue Response ZK path:
>> /overseer/collection-queue-work/qnr-0000000014 doesn't exist. Requestor
>> may have disconnected from ZooKeeper
>> 2020-04-08 21:11:16.690 INFO (qtp1099855928-21) [ ]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
>> params={wt=javabin&version=2&key=solr.core.test1_backup_NEW2.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:UPDATE./update.requests&key=solr.core.test1_backup_NEW2.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_backup_NEW2.shard1.replica_n1:UPDATE./update.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_backup_NEW.shard1.replica_n1:UPDATE./update.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:UPDATE./update.requests&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:QUERY./select.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_backup_NEW.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_backup_NEW.shard1.replica_n1:INDEX.sizeInBytes}
>> status=0 QTime=51
>> 2020-04-08 21:11:16.731 INFO (qtp1099855928-24) [ ]
>> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
>> params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
>> status=0 QTime=27
>>
>>
>>
