Mike, I think having a good default value for maximum parallel operations will play a role in not consuming too many resources. Perhaps defaulting to only a single (or other small number based on testing) parallel action(s) per member at a time and allowing users that want better performance to increase that number would be a good start. That should result in performance improvements, but not place increased burden on any specific member. Especially when bootstrapping new members, relance speed may be more valuable than usual, so making it possible to configure on a per rebalance action level would be prefered.
One clarification from my original proposal: regions can already be rebalanced in parallel, depending on the value of resource.manager.threads (which defaults to 1, so no parallelization or regions in the default case). On Thu, Mar 8, 2018 at 11:46 AM, Michael Stolz <mst...@pivotal.io> wrote: > We should be very careful about how much resource we dedicate to > rebalancing. > > One of our competitors rebalances *much* faster than we do, but in doing so > they consume all available resources. > > At one bank that caused significant loss of incoming market data that was > coming in on a multicast feed, which had a severe adverse effect on the > pricing and risk management functions for a period of time. That bank > removed the competitor's product and for several years no distributed > caching was allowed by the chief architect at that bank. Until he left and > a new chief architect was named they didn't use any distributed caching > products. When they DID go back to using them, it pre-dated Geode, so they > used GemFire largely because GemFire does not consume all available > resources while rebalancing. > > I do think we need to improve our rebalancing such that it iterates until > it achieves balance, but not in a way that will consume all available > resources. > > -- > Mike Stolz > > > On Thu, Mar 8, 2018 at 2:25 PM, Nick Reich <nre...@pivotal.io> wrote: > > > Team, > > > > The time required to undertake a rebalance of a geode cluster has often > > been an area for improvement noted by users. Currently, buckets are moved > > one at a time and we propose that creating a system that moved buckets in > > parallel could greatly improve performance for this feature. > > > > Previously, parallelization was implemented for adding redundant copies > of > > buckets to restore redundancy. However, moving buckets is a more > > complicated matter and requires a different approach than restoration of > > redundancy. The reason for this is that members could be potentially both > > be gaining buckets and giving away buckets at the same time. While giving > > away a bucket, that member still has all of the data for the bucket, > until > > the receiving member has fully received the bucket and it can safely be > > removed from the original owner. This means that unless the member has > the > > memory overhead to store all of the buckets it will receive and all the > > buckets it started with, there is potential that parallel moving of > buckets > > could cause the member to run out of memory. > > > > For this reason, we propose a system that does (potentially) several > rounds > > of concurrent bucket moves: > > 1) A set of moves is calculated to improve balance that meet a > requirement > > that no member both receives and gives away a bucket (no member will have > > memory overhead of an existing bucket it is ultimately removing and a new > > bucket). > > 2) Conduct all calculated bucket moves in parallel. Parameters to > throttle > > this process (to prevent taking too many cluster resources, impacting > > performance) should be added, such as only allowing each member to either > > receive or send a maximum number of buckets concurrently. > > 3) If cluster is not yet balanced, perform additional iterations of > > calculating and conducting bucket moves, until balance is achieved or a > > possible maximum iterations is reached. > > Note: in both the existing and proposed system, regions are rebalanced > one > > at a time. > > > > Please let us know if you have feedback on this approach or additional > > ideas that should be considered. > > >
