Could you add another field(s) to your application and use that instead of creating collections/cores? When you execute a search, instead of picking a core, just search a single large core but add in a field which contains some core ID.
-Joe http://www.lovehorsepower.com On Sun, Aug 31, 2014 at 8:23 AM, Mark Miller <markrmil...@gmail.com> wrote: > > > On Aug 31, 2014, at 4:04 AM, Christoph Schmidt < > christoph.schm...@moresophy.de> wrote: > > > > we see at least two problems when scaling to large number of > collections. I would like to ask the community, if they are known and maybe > already addressed in development: > > We have a SolrCloud running with the following numbers: > > - 5 Servers (each 24 CPUs, 128 RAM) > > - 13.000 Collection with 25.000 SolrCores in the Cloud > > The Cloud is working fine, but we see two problems, if we like to scale > further > > 1. Resource consumption of native system threads > > We see that each collection opens at least two threads: one for the > zookeeper (coreZkRegister-1-thread-5154) and one for the searcher > (searcherExecutor-28357-thread-1) > > We will run in "OutOfMemoryError: unable to create new native thread". > Maybe the architecture could be changed here to use thread pools? > > 2. The shutdown and the startup of one server in the SolrCloud > takes 2 hours. So a rolling start is about 10h. For me the problem seems to > be that leader election is "linear". The Overseer does core per core. The > organisation of the cloud is not done parallel or distributed. Is this > already addressed by https://issues.apache.org/jira/browse/SOLR-5473 or > is there more needed? > > 2. No, but it should have been fixed by another issue that will be in 4.10. > > > - Mark > http://about.me/markrmiller
