CREATEALIAS does not work with more than one collection (Error 503: no servers hosting shard)
Hello Solr experts,
For some strange reason, collection alias does not work in my Solr
instance when more than one collection is used. I would appreciate your
help.
# Here is my setup, which is quite simple:
Zookeeper: 3.4.5 (used to upconfig/linkconfig collections and configs
for c1 and c2)
Solr: version 4.4.0, with two collections c1 and c2 (solr.xml included)
created using remote core API calls
# Symptoms:
1. Solr queries to each individual collection works fine:
http://localhost:8983/solr/c1/select?q=*:*
http://localhost:8983/solr/c2/select?q=*:*
2. CREATEALIAS name=cx for c1 or c2 alone (e.g. 1-1 mapping) works fine:
http://localhost:8983/solr/cx/select?q=*:*
3. CREATEALIAS name=cx for c1 and c2 does not work:
# Solr request/response to the collection alias (success):
http://localhost:8983/solr/cx/select?q=*:*
5032*:*no
servers hosting shard: 503
# Solr query using the alias fails with Error 503: "no servers
hosting shard"
curl -s
"http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=cx&collections=c1,c2";
0134
# Solr logs:
3503223 [qtp724646150-11] ERROR org.apache.solr.core.SolrCore ?
org.apache.solr.common.SolrException: no servers hosting shard:
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:149)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
3503224 [qtp724646150-11] INFO org.apache.solr.core.SolrCore ? [c1]
webapp=/solr path=/select params={q=*:*} status=503 QTime=2
# solr.xml
# zookeeper alias (same from solr/cloud UI):
[zk: localhost:2181(CONNECTED) 10] get /myroot/aliases.json
{"collection":{
"cx":"c1,c2"}}
cZxid = 0x110d
ctime = Fri Sep 13 17:25:18 PDT 2013
mZxid = 0x18d1
mtime = Mon Sep 16 16:31:21 PDT 2013
pZxid = 0x110d
cversion = 0
dataVersion = 19
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 119
numChildren = 0
BTW, I've spent a lot of time figuring out how to make zookeeper and
solr work together. The commands are not complex, but making them work
sometimes requires a lot of digging online, to figure out missing jars
for zkCli.sh, etc. I know a lot of things are changing since Solr 4.0,
but I really hope the Solr documentation can be better maintained, so
that people won't have to spend tons of hours figuring out simple steps
(albeit complex under the hood) like this. Thanks!
--
Regards,
HaiXin
=
AIM : tivohtie
Work : 408.914.9835
Mobile : 408.368.9289
Schedule : http://htie-linux/
=
This email and any attachments may contain confidential and privileged material
for the sole use of the intended recipient. Any review, copying, or
distribution of this email (or any attachments) by others is prohibited. If you
are not the intended recipient, please contact the sender immediately and
permanently delete this email and any attachments. No employee or agent of TiVo
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by
email. Binding agreements with TiVo Inc. may only be made by a signed written
agreement.
Updated: CREATEALIAS does not work with more than one collection (Error 503: no servers hosting shard)
Sorry but I've fixed some typos, updated text:
Hello Solr experts,
For some strange reason, collection alias does not work in my Solr
instance when more than one collection is used. I would appreciate your
help.
# Here is my setup, which is quite simple:
Zookeeper: 3.4.5 (used to upconfig/linkconfig collections and configs
for c1 and c2)
Solr: version 4.4.0, with two collections c1 and c2 (solr.xml included)
created using remote core API calls
# Symptoms:
1. Solr queries to each individual collection works fine:
http://localhost:8983/solr/c1/select?q=*:*
http://localhost:8983/solr/c2/select?q=*:*
2. CREATEALIAS name=cx for c1 or c2 alone (e.g. 1-1 mapping) works fine:
http://localhost:8983/solr/cx/select?q=*:*
3. CREATEALIAS name=cx for c1 and c2 does not work:
# Solr request/response to the collection alias (success):
curl -s
"http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=cx&collections=c1,c2";
0134
# Solr query using the alias fails with Error 503: "no servers
hosting shard"
http://localhost:8983/solr/cx/select?q=*:*
5032*:*no
servers hosting shard: 503
# Solr logs:
3503223 [qtp724646150-11] ERROR org.apache.solr.core.SolrCore ?
org.apache.solr.common.SolrException: no servers hosting shard:
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:149)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
3503224 [qtp724646150-11] INFO org.apache.solr.core.SolrCore ? [c1]
webapp=/solr path=/select params={q=*:*} status=503 QTime=2
# solr.xml
# zookeeper alias (same from solr/cloud UI):
[zk: localhost:2181(CONNECTED) 10] get /myroot/aliases.json
{"collection":{
"cx":"c1,c2"}}
cZxid = 0x110d
ctime = Fri Sep 13 17:25:18 PDT 2013
mZxid = 0x18d1
mtime = Mon Sep 16 16:31:21 PDT 2013
pZxid = 0x110d
cversion = 0
dataVersion = 19
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 119
numChildren = 0
BTW, I've spent a lot of time figuring out how to make zookeeper and
solr work together. The commands are not complex, but making them work
sometimes requires a lot of digging online, to figure out missing jars
for zkCli.sh, etc. I know a lot of things are changing since Solr 4.0,
but I really hope the Solr documentation can be better maintained, so
that people won't have to spend tons of hours figuring out simple steps
(albeit complex under the hood) like this. Thanks!
--
Regards,
HaiXin
This email and any attachments may contain confidential and privileged material
for the sole use of the intended recipient. Any review, copying, or
distribution of this email (or any attachments) by others is prohibited. If you
are not the intended recipient, please contact the sender immediately and
permanently delete this email and any attachments. No employee or agent of TiVo
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by
email. Binding agreements with TiVo Inc. may only be made by a signed written
agreement.
SPLITSHARD failure right before publishing the new sub-shards
Hi Solr experts,
I am using Solr 4.4 with ZK 3.4.5, trying to split "shard1" of a
collection named "body". There is only one core on one machine for this
collection. When I call SPLITSHARD to split this collection, Solr is
able to create two sub-shards, but failed with a NPE in SolrCore.java
while publishing the new shards. It seems that either the updateHandler
or its updateLog is null, though they work fine in the original shard:
SolrCore.java
if (cc != null && cc.isZooKeeperAware() &&
Slice.CONSTRUCTION.equals(cd.getCloudDescriptor().getShardState())) {
// set update log to buffer before publishing the core
862: getUpdateHandler().getUpdateLog().bufferUpdates();
cd.getCloudDescriptor().setShardState(null);
cd.getCloudDescriptor().setShardRange(null);
}
Here are the details. Any pointers to aid debugging this issue is
greatly appreciated!
# curl request/response to split the shard:
curl -s
"http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=body&shard=shard1";
5002688org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error
CREATEing SolrCore 'body_shard1_0_replica1': Unable to create core:
body_shard1_0_replica1 Caused by: nullorg.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
SPLTSHARD failed to create subshard leadersSPLTSHARD failed to create subshard
leaders500SPLTSHARD failed to create subshard leadersorg.apache.solr.common.SolrException: SPLTSHARD failed to
create subshard leaders
at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:171)
at
org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:322)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:136)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
500
# Full solr log for the split shard call:
384779 [qtp334936591-10] INFO
org.apache.solr.handler.admin.CollectionsHandler ? Splitting shard :
shard=shard1&action=SPLITSHARD&collection=body
384791 [main-EventThread] INFO org.apache.solr.cloud.DistributedQueue ?
Watcher fired on path: /overseer/collection-queue-work state:
SyncConnected type NodeChildrenChanged
384791 [main-EventThread] INFO org.apache.solr.cloud.DistributedQueue ?
Watcher fired on path: /overseer/collection-queue-work state:
SyncConnected type NodeChildrenChanged
384797 [main-EventThread] INFO org.apache.solr.cloud.Di
Re: SPLITSHARD failure right before publishing the new sub-shards
Never mind. I figured it out. It was due to a NPE on the missing
updateLog in solrconfig.xml. My solrconfig.xml is from an older Solr
release, which doesn't have certain required sections, etc. After adding
them to solrconfig.xml per this official doc, everything started to
work. It'd be great if null checks were there to produce informative
error on SolrCore.java, so as to make it easier to find the root cause.
http://wiki.apache.org/solr/SolrCloud#Required_Config
Regards,
HaiXin
On 09/16/2013 06:44 PM, HaiXin Tie wrote:
Hi Solr experts,
I am using Solr 4.4 with ZK 3.4.5, trying to split "shard1" of a
collection named "body". There is only one core on one machine for
this collection. When I call SPLITSHARD to split this collection, Solr
is able to create two sub-shards, but failed with a NPE in
SolrCore.java while publishing the new shards. It seems that either
the updateHandler or its updateLog is null, though they work fine in
the original shard:
SolrCore.java
if (cc != null && cc.isZooKeeperAware() &&
Slice.CONSTRUCTION.equals(cd.getCloudDescriptor().getShardState())) {
// set update log to buffer before publishing the core
862: getUpdateHandler().getUpdateLog().bufferUpdates();
cd.getCloudDescriptor().setShardState(null);
cd.getCloudDescriptor().setShardRange(null);
}
Here are the details. Any pointers to aid debugging this issue is
greatly appreciated!
# curl request/response to split the shard:
curl -s
"http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=body&shard=shard1";
5002688org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error
CREATEing SolrCore 'body_shard1_0_replica1': Unable to create core:
body_shard1_0_replica1 Caused by: nullorg.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
SPLTSHARD failed to create subshard leadersSPLTSHARD failed to create subshard
leaders500SPLTSHARD failed to create subshard leadersorg.apache.solr.common.SolrException: SPLTSHARD failed to
create subshard leaders
at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:171)
at
org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:322)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:136)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run
Re: Updated: CREATEALIAS does not work with more than one collection (Error 503: no servers hosting shard)
Never mind. I figured it out. It was due to a NPE on the missing updateLog in
solrconfig.xml. My solrconfig.xml is from an older Solr release, which doesn't
have certain required sections, etc. After adding them to solrconfig.xml per
this official doc, everything started to work.
http://wiki.apache.org/solr/SolrCloud#Required_Config
Regards,
HaiXin
On 09/16/2013 04:55 PM, HaiXin Tie wrote:
Sorry but I've fixed some typos, updated text:
Hello Solr experts,
For some strange reason, collection alias does not work in my Solr instance
when more than one collection is used. I would appreciate your help.
# Here is my setup, which is quite simple:
Zookeeper: 3.4.5 (used to upconfig/linkconfig collections and configs for c1
and c2)
Solr: version 4.4.0, with two collections c1 and c2 (solr.xml included) created
using remote core API calls
# Symptoms:
1. Solr queries to each individual collection works fine:
http://localhost:8983/solr/c1/select?q=*:*
http://localhost:8983/solr/c2/select?q=*:*
2. CREATEALIAS name=cx for c1 or c2 alone (e.g. 1-1 mapping) works fine:
http://localhost:8983/solr/cx/select?q=*:*
3. CREATEALIAS name=cx for c1 and c2 does not work:
# Solr request/response to the collection alias (success):
curl -s
"http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=cx&collections=c1,c2";<http://localhost:8983/solr/admin/collections?action=CREATEALIAS&name=cx&collections=c1,c2>
0134
# Solr query using the alias fails with Error 503: "no servers hosting shard"
http://localhost:8983/solr/cx/select?q=*:*
5032*:*no servers hosting shard: 503
# Solr logs:
3503223 [qtp724646150-11] ERROR org.apache.solr.core.SolrCore ?
org.apache.solr.common.SolrException: no servers hosting shard:
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:149)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
3503224 [qtp724646150-11] INFO org.apache.solr.core.SolrCore ? [c1]
webapp=/solr path=/select params={q=*:*} status=503 QTime=2
# solr.xml
# zookeeper alias (same from solr/cloud UI):
[zk: localhost:2181(CONNECTED) 10] get /myroot/aliases.json
{"collection":{
"cx":"c1,c2"}}
cZxid = 0x110d
ctime = Fri Sep 13 17:25:18 PDT 2013
mZxid = 0x18d1
mtime = Mon Sep 16 16:31:21 PDT 2013
pZxid = 0x110d
cversion = 0
dataVersion = 19
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 119
numChildren = 0
BTW, I've spent a lot of time figuring out how to make zookeeper and solr work
together. The commands are not complex, but making them work sometimes requires
a lot of digging online, to figure out missing jars for zkCli.sh, etc. I know a
lot of things are changing since Solr 4.0, but I really hope the Solr
documentation can be better maintained, so that people won't have to spend tons
of hours figuring out simple steps (albeit complex under the hood) like this.
Thanks!
This email and any attachments may contain confidential and privileged material
for the sole use of the intended recipient. Any review, copying, or
distribution of this email (or any attachments) by others is prohibited. If you
are not the intended recipient, please contact the sender immediately and
permanently delete this email and any attachments. No employee or agent of TiVo
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by
email. Binding agreements with TiVo Inc. may only be made by a signed written
agreement.
