Thanks Mark found the TODO in ZkStateReader.java // TODO: - possibly: incremental update rather than reread everything
Was there a patch they provided back to address this? On Tue, Sep 27, 2011 at 9:20 PM, Mark Miller <markrmil...@gmail.com> wrote: > > On Sep 26, 2011, at 11:42 AM, Jamie Johnson wrote: > >> Is there any limitation, be it technical or for sanity reasons, on the >> number of shards that can be part of a solr cloud implementation? > > > The loggly guys ended up hitting a limit somewhere. Essentially, whenever the > cloud state is updated, info is read about each shard to update the state > (from zookeeper). There is a TODO that I put in there that says something > like, "consider updating this incrementally" - usually the data on most > shards has not changed, so no reason to read it all. They implemented that > today in their own code, but we have not yet done this in trunk. What that > places the upper limit at, I don't know - I imagine it takes quite a few > shards before it ends up being too much of a problem - they shard by user I > believe, so lot's of shards. > > > - Mark Miller > lucidimagination.com > 2011.lucene-eurocon.org | Oct 17-20 | Barcelona > > > > > > > > > > >
