Hi All,
I have started learning Hadoop and its related components. I am following a
tutorial on Hadoop Administration on Udemy. As part of the learning
process, i ran the following command:
$ hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jarrandomtextwriter
-Ddfs.replication=1 /user/bigdata/randomtextwriter
Above command created 30 files each of size 1 GB. Then i ran the below
reduce command:
$ yarn jar/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapreduce.input.fileinputformat.split.minsize=268435456\
-Dmapreduce.job.reduces=8 \
/user/bigdata/randomtext \
/user/bigdata/wordcount
After executing the above command, I just thought of killing the
application after some time so i ran 'yarn application -list' first which
listed a lot many applications out of which one was *wordc**ount*. I killed
that particular application using 'yarn application -kill application-id'.
However, when i checked the scheduler, i could see that several
applications were still showing in Pending state so i ran the following
command:
$ for x in $(yarn application -list -appStates ACCEPTED | awk 'NR > 2 {
print $1 }'); do yarn application -kill $x; done
It was killing the applications as I could see the 'Apps Completed' count
was going up but as soon as all the apps got killed, I saw those
applications again getting created. Even if I stop the whole cluster and
start again, the scheduler shows that there are submitted applications in
Pending state.
Here's the content of fair-scheduler.xml:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<allocations>
<queue name="root">
<schedulingPolicy>drf</schedulingPolicy>
<queue name="default">
<schedulingPolicy>drf</schedulingPolicy>
</queue>
</queue>
<queuePlacementPolicy>
<rule name="specified" create="false"/>
<rule name="default" create="true"/>
</queuePlacementPolicy>
</allocations>
This is just a test cluster. I just want to kill the applications/clear
the application queue. Any help will really be appreciated as I am
struggling with it for the last few days.
Regards
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
