Thanks, Shawn I have already raised the jute.buffersize to 5Mb on the zookeeper server side but still hitting the same problem. Should i make any changes on the solr server side for this (client side changes?)
On Tue, Apr 8, 2014 at 9:09 AM, Shawn Heisey <s...@elyograg.org> wrote: > On 4/8/2014 9:48 AM, KNitin wrote: > >> I am running solr cloud 4.3.1 (there is a plan to upgrade to later >> versions but that would take a few months). I noticed a very peculiar solr >> behavior in solr that beyond *2496* cores I am unable to create any more >> >> collections due to this error >> >> *Could not get shard id for core.....* >> >> >> I also noticed in the solr "tree" view that the overseer's collections >> work >> queue gets stuck >> ( /overseer >> collection-queue-work >> qn-0000000360 >> qn-0000000362 >> qn-0000000364) >> >> The test results are as follows. >> >> With 8 shards and 2 replicas, I can create 156 collections (and then hit >> the above error) >> With 4 shards and 2 replicas, I can create 312 collections (and then hit >> the above error) >> With 2 shards and 2 replicas, I can create 624 collections (and then hit >> the above error) >> >> The total no of cores is 2496 in all the above cases. >> >> I am unable to create any more collections after this due to cannot get >> shard id error? >> >> Is this a known bug or is there a work around for this? Is it fixed in >> future releases? >> > > You're probably hitting configuration limits, which are set high enough > for "typical" scalability requirements. Certain things need to be > increased for extreme scalability. I don't know about all of them, so this > is likely an incomplete list: > > One of them, most likely the one involved here, is the maximum size of the > zookeeper database - the jute.maxbuffer system property, which defaults to > one megabyte. Another is the maximum number of threads allowed by the > servlet container. In Jetty, this is the maxThreads parameter. Another is > the various connection and thread pool settings in the ShardHandler config. > > http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#Unsafe+Options > http://wiki.apache.org/solr/SolrConfigXml#Configuration_ > of_Shard_Handlers_for_Distributed_searches > https://cwiki.apache.org/confluence/display/solr/ > Moving+to+the+New+solr.xml+Format > > As usual, I could be entirely incorrect about everything I'm saying. > > Thanks, > Shawn > >
