Hello, I am trying to use the collections API to add in a secondary collection similar to what is described in this article http://heliosearch.org/solrcloud-assigning-nodes-machines/ It works fine on a jetty setup where bootstrap config dir is specified. When I try to run this on a tomcat solr that has the example configs already in zookeeper from an upconfig it fails
http://SERVERADDRESS:8080/solr/admin/collections?action=CREATE&name=realtimecollection&collection.configName=collection1&numShards=1&createNodeSet=SERVERADDRESS:8080_solr ": Unable to create core: realtimecollection_shard1_replica1 Caused by: null" Any ideas why this fails? >From the logs it seems to be trying to create this in the same folder as the solr 4.8 war. Below are the logs I see and the startup config arguments I am passing into tomcat/SOLR. Thanks, Chris export SOLR_HOME=/ebs-data/bin/solr export JAVA_OPTS="$JAVA_OPTS -DzkHost=ZOOKEEPERCLUSTERADDRESS/app/test" export JAVA_OPTS="$JAVA_OPTS -DnumShards=2" export JAVA_OPTS="$JAVA_OPTS -Dcollection.configName=collection1 " export JAVA_OPTS="$JAVA_OPTS -Dsolr.solr.home=/ebs-data/solr/conf " export JAVA_OPTS="$JAVA_OPTS -Dsolr.data.dir=/ebs-data/solr/data " – A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) 176300 [Overseer-91593013188428078-SERVERADDRESS:8080_solr-n_0000000000] INFO org.apache.solr.cloud.OverseerCollectionProcessor – Creating SolrCores for new collection, shardNames [shard1] , replicationFactor : 1 176300 [Overseer-91593013188428078-SERVERADDRESS:8080_solr-n_0000000000] INFO org.apache.solr.cloud.OverseerCollectionProcessor – Creating shard realtimecollection_shard1_replica1 as part of slice shard1 of collection realtimecollection on SERVERADDRESS:8080_solr 178443 [http-bio-8080-exec-13] INFO org.apache.solr.handler.admin.CoreAdminHandler – core create command numShards=1&shard=shard1&collection.configName=collection1&name=realtimecollection_shard1_replica1&action=CREATE&collection=realtimecollection&wt=javabin&qt=/admin/cores&version=2 178540 [http-bio-8080-exec-13] INFO org.apache.solr.cloud.ZkController – publishing core=realtimecollection_shard1_replica1 state=down collection=realtimecollection 178568 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.DistributedQueue – LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged 178569 [http-bio-8080-exec-13] INFO org.apache.solr.cloud.ZkController – waiting to find shard id in clusterstate for realtimecollection_shard1_replica1 178641 [Thread-2] INFO org.apache.solr.common.cloud.ZkStateReader – Updating cloud state from ZooKeeper... 178667 [Thread-2] INFO org.apache.solr.cloud.Overseer – Update state numShards=1 message={ "operation":"state", "state":"down", "base_url":"http://SERVERADDRESS:8080/solr";, "core":"realtimecollection_shard1_replica1", "roles":null, "node_name":"SERVERADDRESS:8080_solr", "shard":"shard1", "collection":"realtimecollection", "numShards":"1", "core_node_name":"SERVERADDRESS:8080_solr_realtimecollection_shard1_replica1"} 178771 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.DistributedQueue – LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged 178798 [localhost-startStop-1-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader – A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) 179572 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrXMLCoresLocator – Persisted core descriptions to /etc/solr/solr.xml 179573 [http-bio-8080-exec-13] INFO org.apache.solr.cloud.ZkController – Check for collection zkNode:realtimecollection 179600 [http-bio-8080-exec-13] INFO org.apache.solr.cloud.ZkController – Collection zkNode exists 179600 [http-bio-8080-exec-13] INFO org.apache.solr.common.cloud.ZkStateReader – Load collection config from:/collections/realtimecollection 179654 [http-bio-8080-exec-13] INFO org.apache.solr.common.cloud.ZkStateReader – path=/collections/realtimecollection configName=collection1 specified config exists in ZooKeeper 179655 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrResourceLoader – new SolrResourceLoader for directory: '/etc/solr/realtimecollection_shard1_replica1/' 179900 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrConfig – Adding specified lib dirs to ClassLoader 179901 [http-bio-8080-exec-13] WARN org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to add to classloader: ../../../contrib/extraction/lib (resolved as: /etc/solr/realtimecollection_shard1_replica1/../../../contrib/extraction/lib). 179902 [http-bio-8080-exec-13] WARN org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /etc/solr/realtimecollection_shard1_replica1/../../../dist). 179902 [http-bio-8080-exec-13] WARN org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to add to classloader: ../../../contrib/clustering/lib/ (resolved as: /etc/solr/realtimecollection_shard1_replica1/../../../contrib/clustering/lib). 179903 [http-bio-8080-exec-13] WARN org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /etc/solr/realtimecollection_shard1_replica1/../../../dist). 179903 [http-bio-8080-exec-13] WARN org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to add to classloader: ../../../contrib/langid/lib/ (resolved as: /etc/solr/realtimecollection_shard1_replica1/../../../contrib/langid/lib). 179904 [http-bio-8080-exec-13] WARN org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /etc/solr/realtimecollection_shard1_replica1/../../../dist). 179904 [http-bio-8080-exec-13] WARN org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to add to classloader: ../../../contrib/velocity/lib (resolved as: /etc/solr/realtimecollection_shard1_replica1/../../../contrib/velocity/lib). 179904 [http-bio-8080-exec-13] WARN org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /etc/solr/realtimecollection_shard1_replica1/../../../dist). 180011 [http-bio-8080-exec-13] INFO org.apache.solr.update.SolrIndexConfig – IndexWriter infoStream solr logging is enabled 180038 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrConfig – Using Lucene MatchVersion: LUCENE_48 180185 [http-bio-8080-exec-13] INFO org.apache.solr.core.Config – Loaded SolrConfig: solrconfig.xml 180285 [http-bio-8080-exec-13] INFO org.apache.solr.schema.IndexSchema – Reading Solr Schema from schema.xml 180524 [http-bio-8080-exec-13] INFO org.apache.solr.schema.IndexSchema – [realtimecollection_shard1_replica1] Schema name=example 182144 [http-bio-8080-exec-13] INFO org.apache.solr.schema.IndexSchema – unique key field: id 182147 [http-bio-8080-exec-13] INFO org.apache.solr.rest.RestManager – Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english 182148 [http-bio-8080-exec-13] INFO org.apache.solr.rest.RestManager – Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english 183114 [http-bio-8080-exec-13] INFO org.apache.solr.schema.FileExchangeRateProvider – Reloading exchange rates from file currency.xml 183175 [http-bio-8080-exec-13] INFO org.apache.solr.schema.FileExchangeRateProvider – Reloading exchange rates from file currency.xml 185157 [http-bio-8080-exec-13] INFO org.apache.solr.core.CoreContainer – Creating SolrCore 'realtimecollection_shard1_replica1' using configuration from collection realtimecollection 185158 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – solr.NRTCachingDirectoryFactory 185158 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – [realtimecollection_shard1_replica1] Opening new SolrCore at /etc/solr/realtimecollection_shard1_replica1/, dataDir=/ebs-data/solr/data/ 185159 [http-bio-8080-exec-13] INFO org.apache.solr.core.JmxMonitoredMap – JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1de10458 185160 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – [realtimecollection_shard1_replica1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]} 185160 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – [realtimecollection_shard1_replica1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]} 185161 [http-bio-8080-exec-13] INFO org.apache.solr.core.CachingDirectoryFactory – return new directory for /ebs-data/solr/data 185162 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – New index directory detected: old=null new=/ebs-data/solr/data/index/ 185163 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – created json: solr.JSONResponseWriter 185163 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – adding lazy queryResponseWriter: solr.VelocityResponseWriter 185164 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – created velocity: solr.VelocityResponseWriter 185164 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – created xslt: solr.XSLTResponseWriter 185164 [http-bio-8080-exec-13] INFO org.apache.solr.response.XSLTResponseWriter – xsltCacheLifetimeSeconds=5 185170 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – no updateRequestProcessorChain defined as default, creating implicit default 185171 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /select: solr.SearchHandler 185173 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /query: solr.SearchHandler 185174 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /get: solr.RealTimeGetHandler 185175 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /browse: solr.SearchHandler 185177 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /update: solr.UpdateRequestHandler 185178 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /update/json: solr.UpdateRequestHandler 185179 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /update/csv: solr.UpdateRequestHandler 185179 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – adding lazy requestHandler: solr.extraction.ExtractingRequestHandler 185180 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /update/extract: solr.extraction.ExtractingRequestHandler 185180 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – adding lazy requestHandler: solr.FieldAnalysisRequestHandler 185181 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /analysis/field: solr.FieldAnalysisRequestHandler 185181 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – adding lazy requestHandler: solr.DocumentAnalysisRequestHandler 185182 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /analysis/document: solr.DocumentAnalysisRequestHandler 185183 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /admin/: solr.admin.AdminHandlers 185184 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /admin/ping: solr.PingRequestHandler 185185 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /debug/dump: solr.DumpRequestHandler 185186 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /replication: solr.ReplicationHandler 185187 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – adding lazy requestHandler: solr.SearchHandler 185187 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /spell: solr.SearchHandler 185188 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – adding lazy requestHandler: solr.SearchHandler 185188 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /suggest: solr.SearchHandler 185189 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – adding lazy requestHandler: solr.SearchHandler 185213 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /tvrh: solr.SearchHandler 185214 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – adding lazy requestHandler: solr.SearchHandler 185214 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /terms: solr.SearchHandler 185215 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – adding lazy requestHandler: solr.SearchHandler 185215 [http-bio-8080-exec-13] INFO org.apache.solr.core.RequestHandlers – created /elevate: solr.SearchHandler 185217 [http-bio-8080-exec-13] INFO org.apache.solr.handler.loader.XMLLoader – xsltCacheLifetimeSeconds=60 185218 [http-bio-8080-exec-13] INFO org.apache.solr.handler.loader.XMLLoader – xsltCacheLifetimeSeconds=60 185218 [http-bio-8080-exec-13] INFO org.apache.solr.handler.loader.XMLLoader – xsltCacheLifetimeSeconds=60 185223 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – Hard AutoCommit: if uncommited for 15000ms; 185224 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – Soft AutoCommit: disabled 185225 [http-bio-8080-exec-13] INFO org.apache.solr.core.CachingDirectoryFactory – return new directory for /ebs-data/solr/data/index 186227 [http-bio-8080-exec-13] INFO org.apache.solr.core.CachingDirectoryFactory – looking to close /ebs-data/solr/data/index [CachedDir<<refCount=0;path=/ebs-data/solr/data/index;done=true>>] 186228 [http-bio-8080-exec-13] INFO org.apache.solr.core.CachingDirectoryFactory – Closing directory: /ebs-data/solr/data/index 186236 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – [realtimecollection_shard1_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore@810a57c 186243 [http-bio-8080-exec-13] INFO org.apache.solr.update.UpdateHandler – closing DirectUpdateHandler2{commits=0,autocommit maxTime=15000ms,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0} 186243 [http-bio-8080-exec-13] INFO org.apache.solr.update.SolrCoreState – Closing SolrCoreState 186244 [http-bio-8080-exec-13] INFO org.apache.solr.update.DefaultSolrCoreState – SolrCoreState ref count has reached 0 - closing IndexWriter 186244 [http-bio-8080-exec-13] INFO org.apache.solr.update.DefaultSolrCoreState – closing IndexWriter with IndexWriterCloser 186245 [http-bio-8080-exec-13] INFO org.apache.solr.core.SolrCore – [realtimecollection_shard1_replica1] Closing main searcher on request. 186246 [http-bio-8080-exec-13] INFO org.apache.solr.core.CachingDirectoryFactory – Closing NRTCachingDirectoryFactory - 1 directories currently being tracked 186247 [http-bio-8080-exec-13] INFO org.apache.solr.core.CachingDirectoryFactory – looking to close /ebs-data/solr/data [CachedDir<<refCount=0;path=/ebs-data/solr/data;done=false>>] 186248 [http-bio-8080-exec-13] INFO org.apache.solr.core.CachingDirectoryFactory – Closing directory: /ebs-data/solr/data 186252 [http-bio-8080-exec-13] ERROR org.apache.solr.core.CoreContainer – Unable to create core: realtimecollection_shard1_replica1 org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.<init>(SolrCore.java:858) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:641) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:556) at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:571) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:200) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:186) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:726) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:258) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1550) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1662) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:838) ... 24 more Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/ebs-data/solr/data/index/write.lock: java.nio.channels.OverlappingFileLockException at org.apache.lucene.store.Lock.obtain(Lock.java:89) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:710) at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77) at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64) at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267) at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1513) ... 26 more Caused by: java.nio.channels.OverlappingFileLockException at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255) at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152) at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1017) at java.nio.channels.FileChannel.tryLock(FileChannel.java:1154) at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:148) at org.apache.lucene.store.Lock.obtain(Lock.java:100) ... 32 more 186334 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.DistributedQueue – LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged 186336 [http-bio-8080-exec-13] ERROR org.apache.solr.core.SolrCore – org.apache.solr.common.SolrException: Error CREATEing SolrCore 'realtimecollection_shard1_replica1': Unable to create core: realtimecollection_shard1_replica1 Caused by: null at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:608) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:200) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:186) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:726) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:258) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: org.apache.solr.common.SolrException: Unable to create core: realtimecollection_shard1_replica1 at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:911) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:568) at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:571) ... 21 more Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.<init>(SolrCore.java:858) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:641) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:556) ... 22 more Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1550) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1662) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:838) ... 24 more Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/ebs-data/solr/data/index/write.lock: java.nio.channels.OverlappingFileLockException at org.apache.lucene.store.Lock.obtain(Lock.java:89) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:710) at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77) at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64) at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267) at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1513) ... 26 more Caused by: java.nio.channels.OverlappingFileLockException at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255) at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152) at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1017) at java.nio.channels.FileChannel.tryLock(FileChannel.java:1154) at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:148) at org.apache.lucene.store.Lock.obtain(Lock.java:100) ... 32 more 186337 [http-bio-8080-exec-13] INFO org.apache.solr.servlet.SolrDispatchFilter – [admin] webapp=null path=/admin/cores params={numShards=1&shard=shard1&collection.configName=collection1&name=realtimecollection_shard1_replica1&action=CREATE&collection=realtimecollection&wt=javabin&qt=/admin/cores&version=2} status=400 QTime=7895 186487 [Thread-2] INFO org.apache.solr.common.cloud.ZkStateReader – Updating cloud state from ZooKeeper... 186554 [Overseer-91593013188428078-SERVERADDRESS:8080_solr-n_0000000000] ERROR org.apache.solr.cloud.OverseerCollectionProcessor – Error from shard: http://SERVERADDRESS:8080/solr org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'realtimecollection_shard1_replica1': Unable to create core: realtimecollection_shard1_replica1 Caused by: null at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:157) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) 186555 [Overseer-91593013188428078-SERVERADDRESS:8080_solr-n_0000000000] INFO org.apache.solr.cloud.OverseerCollectionProcessor – Finished create command on all shards for collection: realtimecollection 186698 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.DistributedQueue – LatchChildWatcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged 186787 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.DistributedQueue – LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged 186792 [http-bio-8080-exec-9] INFO org.apache.solr.servlet.SolrDispatchFilter – [admin] webapp=null path=/admin/collections params={numShards=1&collection.configName=collection1&createNodeSet=SERVERADDRESS:8080_solr&name=realtimecollection&action=CREATE} status=0 QTime=11206 186795 [Overseer-91593013188428078-SERVERADDRESS:8080_solr-n_0000000000] INFO org.apache.solr.cloud.OverseerCollectionProcessor – Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{failure={null=org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error CREATEing SolrCore 'realtimecollection_shard1_replica1': Unable to create core: realtimecollection_shard1_replica1 Caused by: null}} 186842 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.DistributedQueue – LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged 186908 [localhost-startStop-1-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader – A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
