there are known perf issues in computing very large clusters
give it a try with the following rules
"FOO_CUSTOMER":[
{
"replica":"0",
"sysprop.HELM_CHART":"!FOO_CUSTOMER",
"strict":"true"},
{
"replica":"<2",
"node":"#ANY",
"strict":"false"}]
On Wed, Sep 4, 2019 at 1:49 AM Andrew Kettmann
<andrew.kettm...@evolve24.com> wrote:
>
> Currently our 7.7.2 cluster has ~600 hosts and each collection is using an
> autoscaling policy based on system property. Our goal is a single core per
> host (container, running on K8S). However as we have rolled more
> containers/collections into the cluster any creation/move actions are taking
> a huge amount of time. In fact we generally hit the 180 second timeout if we
> don't schedule it as async. Though the action gets completed anyway. Looking
> at the code, it looks like for each core it is considering the entire cluster.
>
> Right now our autoscaling policies look like this, note we are feeding a
> sysprop on startup for each collection to map to specific containers:
>
> "FOO_CUSTOMER":[
> {
> "replica":"#ALL",
> "sysprop.HELM_CHART":"FOO_CUSTOMER",
> "strict":"true"},
> {
> "replica":"<2",
> "node":"#ANY",
> "strict":"false"}]
>
> Does name based filtering allow wildcards ? Also would that likely fix the
> issue of the time it takes for Solr to decide where cores can go? Or any
> other suggestions for making this more efficient on the Solr overseer? We do
> have dedicated overseer nodes, but the leader maxes out CPU for awhile while
> it is thinking about this.
>
> We are considering putting each collection into its own zookeeper
> znode/chroot if we can't support this many nodes per overseer. I would like
> to avoid that if possible, but also creating a collection in sub 10 minutes
> would be neat too.
>
> I appreciate any input/suggestions anyone has!
>
> [https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com>
> Andrew Kettmann
> DevOps Engineer
> P: 1.314.596.2836
> [LinkedIn]<https://linkedin.com/company/evolve24> [Twitter]
> <https://twitter.com/evolve24> [Instagram]
> <https://www.instagram.com/evolve_24>
>
> evolve24 Confidential & Proprietary Statement: This email and any attachments
> are confidential and may contain information that is privileged, confidential
> or exempt from disclosure under applicable law. It is intended for the use of
> the recipients. If you are not the intended recipient, or believe that you
> have received this communication in error, please do not read, print, copy,
> retransmit, disseminate, or otherwise use the information. Please delete this
> email and attachments, without reading, printing, copying, forwarding or
> saving them, and notify the Sender immediately by reply email. No
> confidentiality or privilege is waived or lost by any transmission in error.
--
-----------------------------------------------------
Noble Paul
