After some additional debugging, I think that this issue is caused by a
possible race condition introduced to ZkController in Solr-5.0.0.
My concerns are around unregister(...) function in ZkController.
In the current code, all cores are traversed and if one of the cores is
using configLocation, configLocationis variable is cleared so that its not
removed from confDirectoryListeners. A possible issue can occur if, after
the list of cores is fetched, a new core is added. If this new core will
use the same config, then traversing "all cores" will not find that the
configuration is used by another core, and it will be removed from
confDirectoryListeners even though its still needed.
In addition, when adding a watch to configuration in watchZKConfDir(..)
function, no lock is used on confDirectoryListeners like in any other place
where this map is accessed.
A possible solution for this issue:
- Add "synchronized (confDirectoryListeners)" to watchZKConfDir(..).
- In unregister(...) function, traverse the list of cores twice. Before the
first loop, obtain a lock on confDirectoryListeners, then look if any core
is using configLocation, then remove configLocation from
confDirectoryListeners if needed. Then the lock should be released. The
second loop will be used for the rest of the code.
I will be glad for any input, is this a real issue or did i miss something?
Is the suggested solution valid?
Thanks,
Bar
2015-04-01 18:16 GMT+03:00 Bar Weiner <weiner....@gmail.com>:
> Hi,
>
> I'm working on upgrading a project from solr-4.10.3 to solr-5.0.0.
> As part of our JUnit tests we have a few tests for deleting/creating
> collections. Each test create&delete a collection with a different name,
> but they all share the same config in ZK.
> When running these tests in Eclipse everything works fine, but when
> running the same tests through Maven we get the following error so I
> suspect this is a timing related issue :
>
> INFO org.apache.solr.rest.ManagedResourceStorage – Setting up
> ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/SIMPLE_CONFIG
> INFO org.apache.solr.rest.ManagedResourceStorage – Configured
> ZooKeeperStorageIO with znodeBase: /configs/SIMPLE_CONFIG
> INFO org.apache.solr.rest.RestManager – Initializing RestManager with
> initArgs: {}
> INFO org.apache.solr.rest.ManagedResourceStorage – Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/SIMPLE_CONFIG
> INFO org.apache.solr.rest.ManagedResourceStorage – No data found for
> znode /configs/SIMPLE_CONFIG/_rest_managed.json
> INFO org.apache.solr.rest.ManagedResourceStorage – Loaded null at path
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/SIMPLE_CONFIG
> INFO org.apache.solr.rest.RestManager – Initializing 0 registered
> ManagedResources
> INFO org.apache.solr.handler.ReplicationHandler – Commits will be
> reserved for 10000
> INFO org.apache.solr.core.SolrCore – [mycollection1] Registered new
> searcher Searcher@3208a6c4[mycollection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> ERROR org.apache.solr.core.CoreContainer – Error creating core
> [mycollection1]: This conf directory is not valid
> org.apache.solr.common.SolrException: This conf directory is not valid
> at
> org.apache.solr.cloud.ZkController.registerConfListenerForCore(ZkController.java:2229)
> at
> org.apache.solr.core.SolrCore.registerConfListener(SolrCore.java:2633)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:936)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:662)
> at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:513)
> at
> org.apache.solr.core.CoreContainer.create(CoreContainer.java:488)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:573)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:197)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:186)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
> at
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:736)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:261)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:204)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:722)
>
> In summary, my question is what is the meaning of this exception and what
> are the possible causes of it. I am sure the config directory I'm using is
> still in ZK and its not deleted during the test.
>
> Thanks,
> Bar
>
>
