Also, is there any easy way upgrading to 4.5 without having to change most of my plugins & configuration files?
On Wed, Oct 16, 2013 at 4:18 PM, Chris <christu...@gmail.com> wrote: > oops, the actual url is -http://64.251.14.47:1981/solr/ > > Also, another issue that needs to be raised is the creation of cores from > the "core admin" section of the gui, doesnt really work well, it creates > files but then they do not work (again i am using 4.4) > > > On Wed, Oct 16, 2013 at 4:12 PM, Chris <christu...@gmail.com> wrote: > >> Hi, >> >> Please find the clusterstate.json as below: >> >> I have created a dev environment on one of my servers so that you can see >> the issue live - http://64.251.14.47:1984/solr/ >> >> Also, There seems to be something wrong in zookeeper, when we try to add >> documents using solrj, it works fine as long as load of insert is not much, >> but once we start doing many inserts, then it throws a lot of errors... >> >> I am doing something like - >> >> CloudSolrServer solrCoreCloud = new CloudSolrServer(cloudURL); >> solrCoreCloud.setDefaultCollection("Image"); >> UpdateResponse up = solrCoreCloud.addBean(resultItem); >> UpdateResponse upr = solrCoreCloud.commit(); >> >> >> >> clusterstate.json --- >> >> { >> "collection1":{ >> "shards":{ >> "shard2":{ >> "range":"b3330000-e665ffff", >> "state":"active", >> "replicas":{"core_node4":{ >> "state":"active", >> "core":"collection1", >> "node_name":"64.251.14.47:1984_solr", >> "base_url":"http://64.251.14.47:1984/solr";, >> "leader":"true"}}}, >> "shard3":{ >> "range":"e6660000-1998ffff", >> "state":"active", >> "replicas":{"core_node5":{ >> "state":"active", >> "core":"collection1", >> "node_name":"64.251.14.47:1985_solr", >> "base_url":"http://64.251.14.47:1985/solr";, >> "leader":"true"}}}, >> "shard4":{ >> "range":"19990000-4ccbffff", >> "state":"active", >> "replicas":{ >> "core_node2":{ >> "state":"active", >> "core":"collection1", >> "node_name":"64.251.14.47:1982_solr", >> "base_url":"http://64.251.14.47:1982/solr"}, >> "core_node6":{ >> "state":"active", >> "core":"collection1", >> "node_name":"64.251.14.47:1981_solr", >> "base_url":"http://64.251.14.47:1981/solr";, >> "leader":"true"}}}, >> "shard5":{ >> "range":"4ccc0000-7fffffff", >> "state":"active", >> "replicas":{"core_node3":{ >> "state":"active", >> "core":"collection1", >> "node_name":"64.251.14.47:1983_solr", >> "base_url":"http://64.251.14.47:1983/solr";, >> "leader":"true"}}}}, >> "router":"compositeId"}, >> "Web":{ >> "shards":{ >> "shard1":{ >> "range":"80000000-b332ffff", >> "state":"active", >> "replicas":{"core_node2":{ >> "state":"active", >> "core":"Web_shard1_replica1", >> "node_name":"64.251.14.47:1983_solr", >> "base_url":"http://64.251.14.47:1983/solr";, >> "leader":"true"}}}, >> "shard2":{ >> "range":"b3330000-e665ffff", >> "state":"active", >> "replicas":{"core_node3":{ >> "state":"active", >> "core":"Web_shard2_replica1", >> "node_name":"64.251.14.47:1984_solr", >> "base_url":"http://64.251.14.47:1984/solr";, >> "leader":"true"}}}, >> "shard3":{ >> "range":"e6660000-1998ffff", >> "state":"active", >> "replicas":{"core_node4":{ >> "state":"active", >> "core":"Web_shard3_replica1", >> "node_name":"64.251.14.47:1982_solr", >> "base_url":"http://64.251.14.47:1982/solr";, >> "leader":"true"}}}, >> "shard4":{ >> "range":"19990000-4ccbffff", >> "state":"active", >> "replicas":{"core_node5":{ >> "state":"active", >> "core":"Web_shard4_replica1", >> "node_name":"64.251.14.47:1985_solr", >> "base_url":"http://64.251.14.47:1985/solr";, >> "leader":"true"}}}, >> "shard5":{ >> "range":null, >> "state":"active", >> "replicas":{"core_node1":{ >> "state":"active", >> "core":"Web_shard5_replica1", >> "node_name":"64.251.14.47:1981_solr", >> "base_url":"http://64.251.14.47:1981/solr";, >> "leader":"true"}}}}, >> "router":"compositeId"}, >> "Image":{ >> "shards":{ >> "shard1":{ >> "range":"80000000-b332ffff", >> "state":"active", >> "replicas":{"core_node1":{ >> "state":"active", >> "core":"Image_shard1_replica1", >> "node_name":"64.251.14.47:1983_solr", >> "base_url":"http://64.251.14.47:1983/solr";, >> "leader":"true"}}}, >> "shard2":{ >> "range":"b3330000-e665ffff", >> "state":"active", >> "replicas":{"core_node2":{ >> "state":"active", >> "core":"Image_shard2_replica1", >> "node_name":"64.251.14.47:1985_solr", >> "base_url":"http://64.251.14.47:1985/solr";, >> "leader":"true"}}}, >> "shard3":{ >> "range":"e6660000-1998ffff", >> "state":"active", >> "replicas":{"core_node3":{ >> "state":"active", >> "core":"Image_shard3_replica1", >> "node_name":"64.251.14.47:1984_solr", >> "base_url":"http://64.251.14.47:1984/solr";, >> "leader":"true"}}}, >> "shard4":{ >> "range":"19990000-4ccbffff", >> "state":"active", >> "replicas":{"core_node5":{ >> "state":"active", >> "core":"Image_shard4_replica1", >> "node_name":"64.251.14.47:1982_solr", >> "base_url":"http://64.251.14.47:1982/solr";, >> "leader":"true"}}}, >> "shard5":{ >> "range":null, >> "state":"active", >> "replicas":{"core_node4":{ >> "state":"active", >> "core":"Image_shard5_replica1", >> "node_name":"64.251.14.47:1981_solr", >> "base_url":"http://64.251.14.47:1981/solr";, >> "leader":"true"}}}}, >> "router":"compositeId"}, >> "image":{ >> "shards":{ >> "shard1":{ >> "range":"80000000-b332ffff", >> "state":"active", >> "replicas":{ >> "core_node1":{ >> "state":"active", >> "core":"image_shard1_replica2", >> "node_name":"64.251.14.47:1985_solr", >> "base_url":"http://64.251.14.47:1985/solr"}, >> "core_node3":{ >> "state":"active", >> "core":"image_shard1_replica1", >> "node_name":"64.251.14.47:1982_solr", >> "base_url":"http://64.251.14.47:1982/solr";, >> "leader":"true"}}}, >> "shard2":{ >> "range":"b3330000-e665ffff", >> "state":"active", >> "replicas":{ >> "core_node9":{ >> "state":"active", >> "core":"image_shard2_replica2", >> "node_name":"64.251.14.47:1984_solr", >> "base_url":"http://64.251.14.47:1984/solr"}, >> "core_node5":{ >> "state":"active", >> "core":"image_shard2_replica1", >> "node_name":"64.251.14.47:1981_solr", >> "base_url":"http://64.251.14.47:1981/solr";, >> "leader":"true"}}}, >> "shard3":{ >> "range":"e6660000-1998ffff", >> "state":"active", >> "replicas":{ >> "core_node4":{ >> "state":"active", >> "core":"image_shard3_replica2", >> "node_name":"64.251.14.47:1982_solr", >> "base_url":"http://64.251.14.47:1982/solr"}, >> "core_node7":{ >> "state":"active", >> "core":"image_shard3_replica1", >> "node_name":"64.251.14.47:1983_solr", >> "base_url":"http://64.251.14.47:1983/solr";, >> "leader":"true"}}}, >> "shard4":{ >> "range":"19990000-4ccbffff", >> "state":"active", >> "replicas":{ >> "core_node2":{ >> "state":"active", >> "core":"image_shard4_replica1", >> "node_name":"64.251.14.47:1985_solr", >> "base_url":"http://64.251.14.47:1985/solr"}, >> "core_node6":{ >> "state":"active", >> "core":"image_shard4_replica2", >> "node_name":"64.251.14.47:1981_solr", >> "base_url":"http://64.251.14.47:1981/solr";, >> "leader":"true"}}}, >> "shard5":{ >> "range":"4ccc0000-7fffffff", >> "state":"active", >> "replicas":{ >> "core_node8":{ >> "state":"active", >> "core":"image_shard5_replica2", >> "node_name":"64.251.14.47:1983_solr", >> "base_url":"http://64.251.14.47:1983/solr"}, >> "core_node10":{ >> "state":"active", >> "core":"image_shard5_replica1", >> "node_name":"64.251.14.47:1984_solr", >> "base_url":"http://64.251.14.47:1984/solr";, >> "leader":"true"}}}}, >> "router":"compositeId"}, >> "Reference":{ >> "shards":{ >> "shard1":{ >> "range":"80000000-b332ffff", >> "state":"active", >> "replicas":{"core_node1":{ >> "state":"active", >> "core":"Reference_shard1_replica1", >> "node_name":"64.251.14.47:1984_solr", >> "base_url":"http://64.251.14.47:1984/solr";, >> "leader":"true"}}}, >> "shard3":{ >> "range":"e6660000-1998ffff", >> "state":"active", >> "replicas":{"core_node2":{ >> "state":"active", >> "core":"Reference_shard3_replica1", >> "node_name":"64.251.14.47:1985_solr", >> "base_url":"http://64.251.14.47:1985/solr";, >> "leader":"true"}}}, >> "shard4":{ >> "range":"19990000-4ccbffff", >> "state":"active", >> "replicas":{"core_node5":{ >> "state":"active", >> "core":"Reference_shard4_replica1", >> "node_name":"64.251.14.47:1982_solr", >> "base_url":"http://64.251.14.47:1982/solr";, >> "leader":"true"}}}, >> "shard5":{ >> "range":"4ccc0000-7fffffff", >> "state":"active", >> "replicas":{"core_node4":{ >> "state":"active", >> "core":"Reference_shard5_replica1", >> "node_name":"64.251.14.47:1983_solr", >> "base_url":"http://64.251.14.47:1983/solr";, >> "leader":"true"}}}, >> "shard2":{ >> "range":null, >> "state":"active", >> "replicas":{"core_node3":{ >> "state":"active", >> "core":"Reference_shard2_replica1", >> "node_name":"64.251.14.47:1981_solr", >> "base_url":"http://64.251.14.47:1981/solr";, >> "leader":"true"}}}}, >> "router":"compositeId"}, >> "web":{ >> "shards":{ >> "shard1":{ >> "range":"80000000-b332ffff", >> "state":"active", >> "replicas":{ >> "core_node4":{ >> "state":"active", >> "core":"web_shard1_replica2", >> "node_name":"64.251.14.47:1983_solr", >> "base_url":"http://64.251.14.47:1983/solr"}, >> "core_node2":{ >> "state":"active", >> "core":"web_shard1_replica1", >> "node_name":"64.251.14.47:1981_solr", >> "base_url":"http://64.251.14.47:1981/solr";, >> "leader":"true"}}}, >> "shard2":{ >> "range":"b3330000-e665ffff", >> "state":"active", >> "replicas":{ >> "core_node6":{ >> "state":"active", >> "core":"web_shard2_replica1", >> "node_name":"64.251.14.47:1982_solr", >> "base_url":"http://64.251.14.47:1982/solr"}, >> "core_node7":{ >> "state":"active", >> "core":"web_shard2_replica2", >> "node_name":"64.251.14.47:1984_solr", >> "base_url":"http://64.251.14.47:1984/solr";, >> "leader":"true"}}}, >> "shard3":{ >> "range":"e6660000-1998ffff", >> "state":"active", >> "replicas":{ >> "core_node9":{ >> "state":"active", >> "core":"web_shard3_replica1", >> "node_name":"64.251.14.47:1985_solr", >> "base_url":"http://64.251.14.47:1985/solr"}, >> "core_node1":{ >> "state":"active", >> "core":"web_shard3_replica2", >> "node_name":"64.251.14.47:1981_solr", >> "base_url":"http://64.251.14.47:1981/solr";, >> "leader":"true"}}}, >> "shard4":{ >> "range":"19990000-4ccbffff", >> "state":"active", >> "replicas":{ >> "core_node3":{ >> "state":"active", >> "core":"web_shard4_replica2", >> "node_name":"64.251.14.47:1982_solr", >> "base_url":"http://64.251.14.47:1982/solr"}, >> "core_node5":{ >> "state":"active", >> "core":"web_shard4_replica1", >> "node_name":"64.251.14.47:1983_solr", >> "base_url":"http://64.251.14.47:1983/solr";, >> "leader":"true"}}}, >> "shard5":{ >> "range":"4ccc0000-7fffffff", >> "state":"active", >> "replicas":{ >> "core_node8":{ >> "state":"active", >> "core":"web_shard5_replica1", >> "node_name":"64.251.14.47:1984_solr", >> "base_url":"http://64.251.14.47:1984/solr"}, >> "core_node10":{ >> "state":"active", >> "core":"web_shard5_replica2", >> "node_name":"64.251.14.47:1985_solr", >> "base_url":"http://64.251.14.47:1985/solr";, >> "leader":"true"}}}}, >> "router":"compositeId"}, >> "News":{ >> "shards":{ >> "shard1":{ >> "range":"80000000-b332ffff", >> "state":"active", >> "replicas":{"core_node1":{ >> "state":"active", >> "core":"News_shard1_replica1", >> "node_name":"64.251.14.47:1984_solr", >> "base_url":"http://64.251.14.47:1984/solr";, >> "leader":"true"}}}, >> "shard2":{ >> "range":"b3330000-e665ffff", >> "state":"active", >> "replicas":{"core_node3":{ >> "state":"active", >> "core":"News_shard2_replica1", >> "node_name":"64.251.14.47:1983_solr", >> "base_url":"http://64.251.14.47:1983/solr";, >> "leader":"true"}}}, >> "shard3":{ >> "range":"e6660000-1998ffff", >> "state":"active", >> "replicas":{"core_node2":{ >> "state":"active", >> "core":"News_shard3_replica1", >> "node_name":"64.251.14.47:1982_solr", >> "base_url":"http://64.251.14.47:1982/solr";, >> "leader":"true"}}}, >> "shard5":{ >> "range":"4ccc0000-7fffffff", >> "state":"active", >> "replicas":{"core_node5":{ >> "state":"active", >> "core":"News_shard5_replica1", >> "node_name":"64.251.14.47:1985_solr", >> "base_url":"http://64.251.14.47:1985/solr";, >> "leader":"true"}}}, >> "shard4":{ >> "range":null, >> "state":"active", >> "replicas":{"core_node4":{ >> "state":"active", >> "core":"News_shard4_replica1", >> "node_name":"64.251.14.47:1981_solr", >> "base_url":"http://64.251.14.47:1981/solr";, >> "leader":"true"}}}}, >> "router":"compositeId"}} >> >> >> >> >> On Wed, Oct 16, 2013 at 2:19 PM, <primoz.sk...@policija.si> wrote: >> >>> Yap, you are right - I only created extra replicas with cores API. For a >>> new shard I had to use "split shard" command. >>> >>> My apologies. >>> >>> Primož >>> >>> >>> >>> From: Shalin Shekhar Mangar <shalinman...@gmail.com> >>> To: solr-user@lucene.apache.org >>> Date: 16.10.2013 10:45 >>> Subject: Re: Regarding Solr Cloud issue... >>> >>> >>> >>> If the initial collection was created with a numShards parameter (and >>> hence >>> compositeId router then there was no way to create a new logical shard. >>> You >>> can add replicas with the core admin API but only to shards that already >>> exist. A new logical shard can only be created by splitting an existing >>> one. >>> >>> The "createshard" API also has the same limitation -- it cannot create a >>> shard for a collection with compositeId router. It is supposed to be used >>> for collections with custom sharding (i.e. "implicit" router). In such >>> collections, there is no concept of a hash range and routing is done >>> explicitly by the user using the "shards" parameter in the request or by >>> sending the request to the target core/node directly. >>> >>> So, in summary, attempting to add a new logical shard to a collection >>> with >>> compositeId router via CoreAdmin APIs is wrong, unsupported and should be >>> disallowed. Adding replicas to existing logical shards is okay though. >>> >>> >>> On Wed, Oct 16, 2013 at 12:56 PM, <primoz.sk...@policija.si> wrote: >>> >>> > If I am not mistaken the only way to create a new shard from a >>> collection >>> > in 4.4.0 was to use cores API. That worked fine for me until I used >>> > *other* cores API commands. Those usually produced null ranges. >>> > >>> > In 4.5.0 this is fixed with newly added commands "createshard" etc. to >>> the >>> > collections API, right? >>> > >>> > Primoz >>> > >>> > >>> > >>> > From: Shalin Shekhar Mangar <shalinman...@gmail.com> >>> > To: solr-user@lucene.apache.org >>> > Date: 16.10.2013 09:06 >>> > Subject: Re: Regarding Solr Cloud issue... >>> > >>> > >>> > >>> > Chris, can you post your complete clusterstate.json? Do all shards have >>> a >>> > null range? Also, did you issue any core admin CREATE commands apart >>> from >>> > the create collection api. >>> > >>> > Primoz, I was able to reproduce this but by doing an illegal operation. >>> > Suppose I create a collection with numShards=5 and then I issue a core >>> > admin create command such as: >>> > >>> > >>> >>> http://localhost:8983/solr/admin/cores?action=CREATE&name=xyz&collection=mycollection51&shard=shard6 >>> >>> > >>> > >>> > Then a "shard6" is added to the collection with a null range. This is a >>> > bug >>> > because we should never allow such a core admin create to succeed >>> anyway. >>> > I'll open an issue. >>> > >>> > >>> > >>> > On Wed, Oct 16, 2013 at 11:49 AM, <primoz.sk...@policija.si> wrote: >>> > >>> > > I sometimes also do get null ranges when doing colletions/cores API >>> > > actions CREATE or/and UNLOAD, etc... In 4.4.0 that was not easily >>> fixed >>> > > because zkCli had problems with "putfile" command, but in 4.5.0 it >>> works >>> > > OK. All you have to do is "download" clusterstate.json from ZK ("get >>> > > /clusterstate.json"), fix ranges to appropriate values and upload the >>> > file >>> > > back to ZK with zkCli. >>> > > >>> > > But why those null ranges happen at all is beyond me :) >>> > > >>> > > Primoz >>> > > >>> > > >>> > > >>> > > From: Shalin Shekhar Mangar <shalinman...@gmail.com> >>> > > To: solr-user@lucene.apache.org >>> > > Date: 16.10.2013 07:37 >>> > > Subject: Re: Regarding Solr Cloud issue... >>> > > >>> > > >>> > > >>> > > I'm sorry I am not able to reproduce this issue. >>> > > >>> > > I started 5 solr-4.4 instances. >>> > > I copied example directory into example1, example2, example3 and >>> > example4 >>> > > cd example; java -Dbootstrap_confdir=./solr/collection1/conf >>> > > -Dcollection.configName=myconf -DzkRun -DnumShards=1 -jar start.jar >>> > > cd example1; java -Djetty.port=7574 -DzkHost=localhost:9983 -jar >>> > start.jar >>> > > cd example2; java -Djetty.port=7575 -DzkHost=localhost:9983 -jar >>> > start.jar >>> > > cd example3; java -Djetty.port=7576 -DzkHost=localhost:9983 -jar >>> > start.jar >>> > > cd example4; java -Djetty.port=7577 -DzkHost=localhost:9983 -jar >>> > start.jar >>> > > >>> > > After that I invoked: >>> > > >>> > > >>> > >>> > >>> >>> http://localhost:8983/solr/admin/collections?action=CREATE&name=mycollection51&numShards=5&replicationFactor=1 >>> >>> > >>> > > >>> > > >>> > > I can see all shards having non-null ranges in clusterstate. >>> > > >>> > > >>> > > On Tue, Oct 15, 2013 at 8:47 PM, Chris <christu...@gmail.com> wrote: >>> > > >>> > > > Hi Shalin,. >>> > > > >>> > > > Thank you for your quick reply. I appreciate all the help. >>> > > > >>> > > > I started the solr cloud servers first...with 5 nodes. >>> > > > >>> > > > then i issued a command like below to create the shards - >>> > > > >>> > > > >>> > > > >>> > > >>> > > >>> > >>> > >>> >>> http://localhost:8983/solr/admin/collections?action=CREATE&name=mycollection&numShards=5&replicationFactor=1 >>> >>> > >>> > > >>> > > > < >>> > > > >>> > > >>> > > >>> > >>> > >>> >>> http://localhost:8983/solr/admin/collections?action=CREATE&name=mycollection&numShards=3&replicationFactor=4 >>> >>> > >>> > > >>> > > > > >>> > > > >>> > > > Please advice. >>> > > > >>> > > > Regards, >>> > > > Chris >>> > > > >>> > > > >>> > > > On Tue, Oct 15, 2013 at 8:07 PM, Shalin Shekhar Mangar < >>> > > > shalinman...@gmail.com> wrote: >>> > > > >>> > > > > How did you create these shards? Can you tell us how to reproduce >>> > the >>> > > > > issue? >>> > > > > >>> > > > > Any shard in a collection with compositeId router should never >>> have >>> > > null >>> > > > > ranges. >>> > > > > >>> > > > > >>> > > > > On Tue, Oct 15, 2013 at 7:07 PM, Chris <christu...@gmail.com> >>> wrote: >>> > > > > >>> > > > > > Hi, >>> > > > > > >>> > > > > > I am using solr 4.4 as cloud. while creating shards i see that >>> the >>> > > last >>> > > > > > shard has range of "null". i am not sure if this is a bug. >>> > > > > > >>> > > > > > I am stuck with having null value for the range in >>> > clusterstate.json >>> > > > > > (attached below) >>> > > > > > >>> > > > > > "shard5":{ "range":null, "state":"active", >>> > > "replicas":{"core_node1":{ >>> > > > > > "state":"active", "core":"Web_shard5_replica1", >>> > > > > > "node_name":"domain-name.com:1981_solr", "base_url":" >>> > > > > > http://domain-name.com:1981/solr";, "leader":"true"}}}}, >>> > > > > > "router":"compositeId"}, >>> > > > > > >>> > > > > > I tried to use zookeeper cli to change this, but it was not >>> able >>> > to. >>> > > I >>> > > > > > tried to locate this file, but didn't find it anywhere. >>> > > > > > >>> > > > > > Can you please let me know how do i change the range from null >>> to >>> > > > > something >>> > > > > > meaningful? i have the range that i need, so if i can find the >>> > file, >>> > > > > maybe >>> > > > > > i can change it manually. >>> > > > > > >>> > > > > > My next question is - can we have a catch all for ranges, i >>> mean >>> > if >>> > > > > things >>> > > > > > don't match any other range then insert in this shard..is this >>> > > > possible? >>> > > > > > >>> > > > > > Kindly advice. >>> > > > > > Chris >>> > > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > Regards, >>> > > > > Shalin Shekhar Mangar. >>> > > > > >>> > > > >>> > > >>> > > >>> > > >>> > > -- >>> > > Regards, >>> > > Shalin Shekhar Mangar. >>> > > >>> > > >>> > >>> > >>> > -- >>> > Regards, >>> > Shalin Shekhar Mangar. >>> > >>> > >>> >>> >>> -- >>> Regards, >>> Shalin Shekhar Mangar. >>> >>> >> >
