Yesterday I restarted a development solrcloud. After the cloud restarted 2 collections failed to come back.
I see this in the log: 2018-02-16 15:31:16.684 ERROR (coreLoadExecutor-6-thread-1-processing-n:ae1c-ecomdev-msc02:8983_solr) [ ] o.a.s.c.CachingDirectoryFactory Error closing directory:org.apache.solr.common.SolrException: Timeout waiting for all directory ref counts to be released - gave up waiting on CachedDir<<refCount=1;path=/var/solr/data/bb-catalog-material_shard1_replica1/data/index;done=false>> at org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:178) at org.apache.solr.core.SolrCore.close(SolrCore.java:1633) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1000) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1037) at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:642) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) right after this I also see the following error: 2018-02-16 15:31:16.713 ERROR (coreContainerWorkExecutor-2-thread-1-processing-n:ae1c-ecomdev-msc02:8983_solr) [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be created java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [bb-catalog-material_shard1_replica1] at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.solr.core.CoreContainer.lambda$load$14(CoreContainer.java:671) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.common.SolrException: Unable to create core [bb-catalog-material_shard1_replica1] at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1053) at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:642) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) ... 5 more Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1008) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1037) ... 7 more Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2095) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2215) at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1091) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:980) ... 9 more Caused by: java.lang.IllegalArgumentException: cannot change DocValues type from SORTED to SORTED_SET for field "search_vendor_sku" at org.apache.lucene.index.FieldInfo.setDocValuesType(FieldInfo.java:212) at org.apache.lucene.index.FieldInfos$Builder.addOrUpdateInternal(FieldInfos.java:438) at org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:451) at org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:446) at org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:380) at org.apache.lucene.index.MultiFields.getMergedFieldInfos(MultiFields.java:268) at org.apache.solr.index.SlowCompositeReaderWrapper.getFieldInfos(SlowCompositeReaderWrapper.java:263) at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:279) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2063) We have not modified the field definition for search_vendor_sku <field name="display_vendor_sku" type="string" indexed="false" stored="true" required="false" multiValued="false" docValues="true"/><field name="search_vendor_sku" type="text_lowercase" indexed="true" stored="true" required="false" multiValued="false" docValues="false"/> Both collections had the same error A third collection with the same configuration restarted with no problems. My theory is that the field was changed from a string to text_lowercase, and docValues set to false then data was reloaded. No issues on the reload, but maybe there were some docValues data left over from its previous incarnation as a string? This is dev, I just want to understand what happened so we can prevent it in production -- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does not accept liability for any damages caused by any virus transmitted therewith. Click http://www.emdgroup.com/disclaimer to access the German, French, Spanish and Portuguese versions of this disclaimer.
