Re: Preemption make all applications takes much more time to finish

[Lin Zhao]

Wow. I've been in this group for years and this is the first post in like 
forever.

So if you are still early in your project, run! Run and don't look back. Hadoop 
is dead. Go after the shiny ones. Spark, Storm, Akka etc. etc.

On May 15, 2017 10:21 PM, wuchang <583424...@qq.com> wrote:
My cluster has 9 machines with total memory 643G and 288 vCores.I use 
fair-scheduler  as my scheduler.
Before I turn preemption on , my fair-scheduler.xml is :


<allocations>
    <queue name="highPriority">
       <minResources>100000 mb, 30 vcores</minResources>
       <maxResources>250000 mb, 100 vcores</maxResources>
    </queue>
    <queue name="default">
       <minResources>50000 mb, 20 vcores</minResources>
       <maxResources>100000 mb, 50 vcores</maxResources>
       <maxAMShare>-1.0f</maxAMShare>
    </queue>
    <queue name="ep">
       <minResources>100000 mb, 30 vcores</minResources>
       <maxResources>300000 mb, 100 vcores</maxResources>
       <maxAMShare>-1.0f</maxAMShare>
    </queue>
    <queue name="vip">
       <minResources>30000 mb, 20 vcores</minResources>
       <maxResources>60000 mb, 50 vcores</maxResources>
       <maxAMShare>-1.0f</maxAMShare>
     </queue>
  <fairSharePreemptionTimeout>300</fairSharePreemptionTimeout>
</allocations>

The yarn-site.xml is:

        <property>
                <name>yarn.scheduler.fair.preemption</name>
                <value>false</value>
        </property>


The sum of maxResources of each queue is equal to the total memory of the 
cluster. Of course ,due to the limitation of  maxResources

  for each queue , when a queue if full and other queues is idle , it still 
cannot get any resources from other queues.So , the total resource usage of my 
cluster is always low.


So I turn the preemption on and change the fair-scheduler.xml to this:



<allocations>
    <queue name="highPriority">
       <minResources>100000 mb, 30 vcores</minResources>
       <maxResources>500000 mb, 100 vcores</maxResources>
       <weight>0.35</weight>
       <minSharePreemptionTimeout>20</minSharePreemptionTimeout>
       <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout>
       <fairSharePreemptionThreshold>0.8</fairSharePreemptionThreshold>
    </queue>
    <queue name="default">
       <minResources>25000 mb, 20 vcores</minResources>
       <maxResources>225000 mb, 70 vcores</maxResources>
       <weight>0.14</weight>
       <minSharePreemptionTimeout>20</minSharePreemptionTimeout>
       <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout>
       <fairSharePreemptionThreshold>0.5</fairSharePreemptionThreshold>
       <maxAMShare>-1.0f</maxAMShare>
    </queue>
    <queue name="ep">
       <minResources>200000 mb, 30 vcores</minResources>
       <maxResources>600000 mb, 100 vcores</maxResources>
       <weight>0.42</weight>
       <minSharePreemptionTimeout>20</minSharePreemptionTimeout>
       <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout>
       <fairSharePreemptionThreshold>0.8</fairSharePreemptionThreshold>
       <maxAMShare>-1.0f</maxAMShare>
    </queue>
    <queue name="vip">
       <minResources>50000 mb, 20 vcores</minResources>
       <maxResources>120000 mb, 30 vcores</maxResources>
       <weight>0.09</weight>
       <minSharePreemptionTimeout>20</minSharePreemptionTimeout>
       <fairSharePreemptionTimeout>25</fairSharePreemptionTimeout>
       <fairSharePreemptionThreshold>0.8</fairSharePreemptionThreshold>
       <maxAMShare>-1.0f</maxAMShare>
     </queue>
</allocations>


ye Y
I double the maxResources for each queue .I know that even preemption is on , 
the total memory of each queue will not excess the maxResources config.
Yes , the cluster resource usage can goes to 90% , it proves that the total 
cluster resource usage improves , but yarn applications always takes much more 
time to finish.I don’t know what happened.
Anyone could give me some suggestions?