yes my job has about 160,000 maps and my cluster not getting fully utilized
around 6000 maps ran for 2 hrs and then I killed the job. At any point of
time only 40 containers are running thats just 11% of my cluster capacity.
{
"classification": "mapred-site",
"properties": {
"mapreduce.job.reduce.slowstart.completedmaps":"1",
"mapreduce.reduce.memory.mb": "3072",
"mapreduce.map.memory.mb": "2208",
"mapreduce.map.java.opts":"-Xmx1800m",
"mapreduce.map.cpu.vcores":"1"
}
},
{
"classification": "yarn-site",
"properties": {
"yarn.scheduler.minimum-allocation-mb": "32”,
“yarn.scheduler.maximum-allocation-mb”:”253952”,
“yarn.scheduler.maximum-allocation-vcores: “128”
"yarn.nodemanager.vmem-pmem-ratio":"3",
"yarn.nodemanager.vmem-check-enabled":"true",
yarn.nodemanager.resource.cpu-vcores" ; "16”,
yarn.nodemanager.resource.memory-mb: “23040"
}
Each node: capacity
Disk-space=100gb
memory=28gb
processors: 8
