Heya, I implemented something similar last year to guarantee resource provisioning when we deployed to YARN. We stuck to one-label-per-node to keep things relatively simple. Iirc, these are the basic steps:
- add `yarn.node-labels.configuration-type=centralized` to your yarn-site.xml - set up your queues with `yarn.scheduler.capacity.root.queues=<queue>,<queue>,<queue>`, `yarn.scheduler.capacity.root.<queue>.queues=<subqueue>,<subqueue>`, etc (we only used one layer) - add provisioning requirements as needed (ie `yarn.scheduler.capacity.root.capacity=100`, `yarn.scheduler.capacity.root.<queue>.capacity=X`, etc) - add accessible labels for the queues, ie `yarn.scheduler.capacity.root.<queue>.accessible-node-labels=<LABEL1>[,<LABEL2>,..]` With all of this organization in place, you can then allocate nodes dynamically using the main `yarn rmadmin` command, ie $HADOOP_YARN_HOME/bin/yarn rmadmin -addToClusterNodeLabels LABEL1,LABEL2,.. $HADOOP_YARN_HOME/bin/yarn rmadmin -replaceLabelsOnNode "node.dns=LABEL node.dns=LABEL ..." Malcolm McFarland Cavulus Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any unauthorized or improper disclosure, copying, distribution, or use of the contents of this message is prohibited. The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If you have received this message in error, please notify the sender immediately and delete the original message. On Thu, May 13, 2021 at 3:05 AM HK <[email protected]> wrote: > > You can automate to create a capacity-scheduler.xml based on the requirement, > after that you can deploy it on RM, and refresh the queue. > Is it your requirement to not to restart RM or not to change capacity > scheduler? > > On Thu, May 13, 2021 at 2:45 PM 慧波彭 <[email protected]> wrote: >> >> Hello, we use capacity scheduler to allocate resources in our production >> environment, and use node label to isolate resources. >> There is a demand that we want to dynamically create node labels and >> associate node labels to existing queue without changing >> capacity-scheduler.xml. >> Does anyone know how to implement it? >> I am looking forward to hearing from you. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
