Even if the cluster dont have enough resources it should connect to "
/0.0.0.0:8030" right? it should connect to my <RM_HOST:8030>, not sure why its trying to connect to 0.0.0.0:8030. I have verified the config and i removed traces of 0.0.0.0 still no luck. org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030 If an one has any clue please share. Thanks, Ram On Fri, Aug 19, 2016 at 2:32 PM, rammohan ganapavarapu < [email protected]> wrote: > When i submit a job using yarn its seems working only with oozie its > failing i guess, not sure what is missing. > > yarn jar > /uap/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar > pi 20 1000 > Number of Maps = 20 > Samples per Map = 1000 > . > . > . > Job Finished in 19.622 seconds > Estimated value of Pi is 3.14280000000000000000 > > Ram > > On Fri, Aug 19, 2016 at 11:46 AM, rammohan ganapavarapu < > [email protected]> wrote: > >> Ok, i have used yarn-utils.py to get the correct values for my cluster >> and update those properties and restarted RM and NM but still no luck not >> sure what i am missing, any other insights will help me. >> >> Below are my properties from yarn-site.xml and map-site.xml. >> >> python yarn-utils.py -c 24 -m 63 -d 3 -k False >> Using cores=24 memory=63GB disks=3 hbase=False >> Profile: cores=24 memory=63488MB reserved=1GB usableMem=62GB disks=3 >> Num Container=6 >> Container Ram=10240MB >> Used Ram=60GB >> Unused Ram=1GB >> yarn.scheduler.minimum-allocation-mb=10240 >> yarn.scheduler.maximum-allocation-mb=61440 >> yarn.nodemanager.resource.memory-mb=61440 >> mapreduce.map.memory.mb=5120 >> mapreduce.map.java.opts=-Xmx4096m >> mapreduce.reduce.memory.mb=10240 >> mapreduce.reduce.java.opts=-Xmx8192m >> yarn.app.mapreduce.am.resource.mb=5120 >> yarn.app.mapreduce.am.command-opts=-Xmx4096m >> mapreduce.task.io.sort.mb=1024 >> >> >> <property> >> <name>mapreduce.map.memory.mb</name> >> <value>5120</value> >> </property> >> <property> >> <name>mapreduce.map.java.opts</name> >> <value>-Xmx4096m</value> >> </property> >> <property> >> <name>mapreduce.reduce.memory.mb</name> >> <value>10240</value> >> </property> >> <property> >> <name>mapreduce.reduce.java.opts</name> >> <value>-Xmx8192m</value> >> </property> >> <property> >> <name>yarn.app.mapreduce.am.resource.mb</name> >> <value>5120</value> >> </property> >> <property> >> <name>yarn.app.mapreduce.am.command-opts</name> >> <value>-Xmx4096m</value> >> </property> >> <property> >> <name>mapreduce.task.io.sort.mb</name> >> <value>1024</value> >> </property> >> >> >> >> <property> >> <name>yarn.scheduler.minimum-allocation-mb</name> >> <value>10240</value> >> </property> >> >> <property> >> <name>yarn.scheduler.maximum-allocation-mb</name> >> <value>61440</value> >> </property> >> >> <property> >> <name>yarn.nodemanager.resource.memory-mb</name> >> <value>61440</value> >> </property> >> >> >> Ram >> >> On Thu, Aug 18, 2016 at 11:14 PM, tkg_cangkul <[email protected]> >> wrote: >> >>> maybe this link can be some reference to tune up the cluster: >>> >>> http://jason4zhu.blogspot.co.id/2014/10/memory-configuration >>> -in-hadoop.html >>> >>> >>> On 19/08/16 11:13, rammohan ganapavarapu wrote: >>> >>> Do you know what properties to tune? >>> >>> Thanks, >>> Ram >>> >>> On Thu, Aug 18, 2016 at 9:11 PM, tkg_cangkul <[email protected]> >>> wrote: >>> >>>> i think that's because you don't have enough resource. u can tune your >>>> cluster config to maximize your resource. >>>> >>>> >>>> On 19/08/16 11:03, rammohan ganapavarapu wrote: >>>> >>>> I dont see any thing odd except this not sure if i have to worry about >>>> it or not. >>>> >>>> 2016-08-19 03:29:26,621 INFO [main] org.apache.hadoop.yarn.client.RMProxy: >>>> Connecting to ResourceManager at /0.0.0.0:8030 >>>> 2016-08-19 03:29:27,646 INFO [main] org.apache.hadoop.ipc.Client: >>>> Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 >>>> time(s); retry policy is RetryUpToMaximumCo >>>> untWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) >>>> 2016-08-19 03:29:28,647 INFO [main] org.apache.hadoop.ipc.Client: >>>> Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 >>>> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, >>>> sleepTime=1000 MILLISECONDS) >>>> >>>> >>>> its keep printing this log ..in app container logs. >>>> >>>> On Thu, Aug 18, 2016 at 8:20 PM, tkg_cangkul <[email protected]> >>>> wrote: >>>> >>>>> maybe u can check the logs from port 8088 on your browser. that was RM >>>>> UI. just choose your job id and then check the logs. >>>>> >>>>> On 19/08/16 10:14, rammohan ganapavarapu wrote: >>>>> >>>>> Sunil, >>>>> >>>>> Thanks you for your input, below are my server metrics for RM. Also >>>>> attached RM UI for capacity scheduler resources. How else i can find? >>>>> >>>>> { >>>>> "name": "Hadoop:service=ResourceManage >>>>> r,name=QueueMetrics,q0=root", >>>>> "modelerType": "QueueMetrics,q0=root", >>>>> "tag.Queue": "root", >>>>> "tag.Context": "yarn", >>>>> "tag.Hostname": "hadoop001", >>>>> "running_0": 0, >>>>> "running_60": 0, >>>>> "running_300": 0, >>>>> "running_1440": 0, >>>>> "AppsSubmitted": 1, >>>>> "AppsRunning": 0, >>>>> "AppsPending": 0, >>>>> "AppsCompleted": 0, >>>>> "AppsKilled": 0, >>>>> "AppsFailed": 1, >>>>> "AllocatedMB": 0, >>>>> "AllocatedVCores": 0, >>>>> "AllocatedContainers": 0, >>>>> "AggregateContainersAllocated": 2, >>>>> "AggregateContainersReleased": 2, >>>>> "AvailableMB": 64512, >>>>> "AvailableVCores": 24, >>>>> "PendingMB": 0, >>>>> "PendingVCores": 0, >>>>> "PendingContainers": 0, >>>>> "ReservedMB": 0, >>>>> "ReservedVCores": 0, >>>>> "ReservedContainers": 0, >>>>> "ActiveUsers": 0, >>>>> "ActiveApplications": 0 >>>>> }, >>>>> >>>>> On Thu, Aug 18, 2016 at 6:49 PM, Sunil Govind <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> It could be because of many of reasons. Also I am not sure about >>>>>> which scheduler your are using, pls share more details such as RM log >>>>>> etc. >>>>>> >>>>>> I could point out few reasons >>>>>> - Such as "Not enough resource is cluster" can cause this >>>>>> - If using Capacity Scheduler, if queue capacity is maxed out, such >>>>>> case can happen. >>>>>> - Similarly if max-am-resource-percent is crossed per queue level, >>>>>> then also AM container may not be launched. >>>>>> >>>>>> you could check RM log to get more information if AM container is >>>>>> laucnhed. >>>>>> >>>>>> Thanks >>>>>> Sunil >>>>>> >>>>>> On Fri, Aug 19, 2016 at 5:37 AM rammohan ganapavarapu < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> When i submit a MR job, i am getting this from AM UI but it never >>>>>>> get finished, what am i missing ? >>>>>>> >>>>>>> Thanks, >>>>>>> Ram >>>>>>> >>>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >
