Hi all,
I installed a basic 3 nodes Hadoop 2.9.1 cluster and playing with YARN
settings.
The 3 nodes has following configuration:
1 cpu / 1 core / 512MB RAM
I wonder I was able to configure yarn-site.xml with following settings
(higher than node physical limits) and successfully run a mapreduce 'pi 1
10' job
quote...
<property>
<name>yarn.resourcemanager.scheduler.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>5120</value>
<description>Amount of physical memory, in MB, that can be
allocated for containers. If set to -1 and
yarn.nodemanager.resource.detect-hardware-capabilities is true, it is
automatically calculated. In other cases, the default is
8192MB</description>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>6</value>
<description>Number of CPU cores that can be allocated for
containers.</description>
</property>
...unquote
Can anyone provide an explanation please?
Should 'yarn.nodemanager.vmem-check-enabled' and
'yarn.nodemanager.pmem-check-enabled' properties (set to 'true' as default)
check that my YARN settings are higher than physical limits?
Which mapreduce 'pi' job settings can I use, to 'force' containers to use
more than node physical resources?
Many thanks in advance!
Guido