Hi , I have cluster of around 30 nodes , where each node is configured to 32 gb memory , with each mapper and reducer to use 4gb memory , in one of the queue(backfill_q) , queue is always underutilized. A simple mapreduce job can max use 44 containers instead of 70 available and this applies to sum of all jobs containers currently running in the queue , means to say yarn can only provide 44 out of 70 containers . Below is our scheduling properties. Any help in which config do I need to look , or some messing I have done with config would be highly appreciated.
Some more info * These are pig jobs * at a time some 2000 containers are requested Let me know if more info is required . Thanks in advance. yarn.scheduler.capacity.maximum-am-resource-percent=0.25 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator yarn.scheduler.capacity.queue-mappings-override.enable=false yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.maximum-capacity=100 yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.queues=air_q,analytics_q,backfill_q,campaigns_q,common_csv_q,default,faq_q,sdk_q yarn.scheduler.capacity.root.accessible-node-labels=campaigns_nl,air_nl,backfill_nl,sdk_nl,common_csv_nl yarn.scheduler.capacity.root.accessible-node-labels.campaigns_nl.capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.campaigns_nl.maximum-capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.air_nl.capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.air_nl.maximum-capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.backfill_nl.capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.backfill_nl.maximum-capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.sdk_nl.capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.sdk_nl.maximum-capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.common_csv_nl.capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.common_csv_nl.maximum-capacity=100 yarn.scheduler.capacity.root.air_q.acl_administer_queue=* yarn.scheduler.capacity.root.air_q.acl_submit_applications=* yarn.scheduler.capacity.root.air_q.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.air_q.maximum-capacity=1 yarn.scheduler.capacity.root.air_q.user-limit-factor=1 yarn.scheduler.capacity.root.air_q.state=RUNNING yarn.scheduler.capacity.root.air_q.capacity=1 yarn.scheduler.capacity.root.air_q.default-node-label-expression=air_nl yarn.scheduler.capacity.root.air_q.ordering-policy=fifo yarn.scheduler.capacity.root.air_q.accessible-node-labels=air_nl yarn.scheduler.capacity.root.air_q.accessible-node-labels.air_nl.capacity=100 yarn.scheduler.capacity.root.air_q.accessible-node-labels.air_nl.maximum-capacity=100 yarn.scheduler.capacity.root.analytics_q.acl_administer_queue=* yarn.scheduler.capacity.root.analytics_q.acl_submit_applications=* yarn.scheduler.capacity.root.analytics_q.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.analytics_q.maximum-capacity=100 yarn.scheduler.capacity.root.analytics_q.user-limit-factor=1 yarn.scheduler.capacity.root.analytics_q.state=RUNNING yarn.scheduler.capacity.root.analytics_q.capacity=0 yarn.scheduler.capacity.root.analytics_q.ordering-policy=fifo yarn.scheduler.capacity.root.backfill_q.acl_administer_queue=* yarn.scheduler.capacity.root.backfill_q.acl_submit_applications=* yarn.scheduler.capacity.root.backfill_q.minimum-user-limit-percent=10 yarn.scheduler.capacity.root.backfill_q.maximum-capacity=10 yarn.scheduler.capacity.root.backfill_q.user-limit-factor=10 yarn.scheduler.capacity.root.backfill_q.state=RUNNING yarn.scheduler.capacity.root.backfill_q.capacity=10 yarn.scheduler.capacity.root.backfill_q.ordering-policy=fair yarn.scheduler.capacity.root.backfill_q.ordering-policy.fair.enable-size-based-weight=false yarn.scheduler.capacity.root.backfill_q.maximum-am-resource-percent=0.2 yarn.scheduler.capacity.root.backfill_q.accessible-node-labels= yarn.scheduler.capacity.root.campaigns_q.acl_administer_queue=* yarn.scheduler.capacity.root.campaigns_q.acl_submit_applications=* yarn.scheduler.capacity.root.campaigns_q.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.campaigns_q.maximum-capacity=100 yarn.scheduler.capacity.root.campaigns_q.user-limit-factor=3 yarn.scheduler.capacity.root.campaigns_q.state=RUNNING yarn.scheduler.capacity.root.campaigns_q.capacity=15 yarn.scheduler.capacity.root.campaigns_q.default-node-label-expression=campaigns_nl yarn.scheduler.capacity.root.campaigns_q.ordering-policy=fifo yarn.scheduler.capacity.root.campaigns_q.accessible-node-labels=campaigns_nl yarn.scheduler.capacity.root.campaigns_q.accessible-node-labels.campaigns_nl.capacity=100 yarn.scheduler.capacity.root.campaigns_q.accessible-node-labels.campaigns_nl.maximum-capacity=100 yarn.scheduler.capacity.root.common_csv_q.acl_administer_queue=* yarn.scheduler.capacity.root.common_csv_q.acl_submit_applications=* yarn.scheduler.capacity.root.common_csv_q.minimum-user-limit-percent=25 yarn.scheduler.capacity.root.common_csv_q.maximum-capacity=100 yarn.scheduler.capacity.root.common_csv_q.user-limit-factor=4 yarn.scheduler.capacity.root.common_csv_q.state=RUNNING yarn.scheduler.capacity.root.common_csv_q.capacity=25 yarn.scheduler.capacity.root.common_csv_q.default-node-label-expression=common_csv_nl yarn.scheduler.capacity.root.common_csv_q.ordering-policy=fifo yarn.scheduler.capacity.root.common_csv_q.accessible-node-labels=common_csv_nl yarn.scheduler.capacity.root.common_csv_q.accessible-node-labels.common_csv_nl.capacity=100 yarn.scheduler.capacity.root.common_csv_q.accessible-node-labels.common_csv_nl.maximum-capacity=100 yarn.scheduler.capacity.root.default.acl_submit_applications=* yarn.scheduler.capacity.root.default.maximum-capacity=100 yarn.scheduler.capacity.root.default.user-limit-factor=1 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.capacity=47 yarn.scheduler.capacity.root.default.ordering-policy=fifo yarn.scheduler.capacity.root.default.maximum-am-resource-percent=0.1 yarn.scheduler.capacity.root.faq_q.acl_administer_queue=* yarn.scheduler.capacity.root.faq_q.acl_submit_applications=* yarn.scheduler.capacity.root.faq_q.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.faq_q.maximum-capacity=1 yarn.scheduler.capacity.root.faq_q.user-limit-factor=1 yarn.scheduler.capacity.root.faq_q.state=RUNNING yarn.scheduler.capacity.root.faq_q.capacity=1 yarn.scheduler.capacity.root.faq_q.default-node-label-expression=sdk_nl yarn.scheduler.capacity.root.faq_q.ordering-policy=fifo yarn.scheduler.capacity.root.faq_q.accessible-node-labels=sdk_nl yarn.scheduler.capacity.root.faq_q.accessible-node-labels.sdk_nl.capacity=25 yarn.scheduler.capacity.root.faq_q.accessible-node-labels.sdk_nl.maximum-capacity=100 yarn.scheduler.capacity.root.sdk_q.acl_administer_queue=* yarn.scheduler.capacity.root.sdk_q.acl_submit_applications=* yarn.scheduler.capacity.root.sdk_q.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.sdk_q.maximum-capacity=1 yarn.scheduler.capacity.root.sdk_q.user-limit-factor=1 yarn.scheduler.capacity.root.sdk_q.state=RUNNING yarn.scheduler.capacity.root.sdk_q.capacity=1 yarn.scheduler.capacity.root.sdk_q.default-node-label-expression=sdk_nl yarn.scheduler.capacity.root.sdk_q.ordering-policy=fifo yarn.scheduler.capacity.root.sdk_q.accessible-node-labels=sdk_nl yarn.scheduler.capacity.root.sdk_q.accessible-node-labels.sdk_nl.capacity=75 yarn.scheduler.capacity.root.sdk_q.accessible-node-labels.sdk_nl.maximum-capacity=100 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
