It's AM log, we need some RM log grep by `1533044360609_6754589`. Thanks! On Fri, May 17, 2019 at 2:41 PM lk_hadoop <[email protected]> wrote:
> I can only find syslog from RM , Please see the E-mail attachments. > > 2019-05-17 > ------------------------------ > lk_hadoop > ------------------------------ > > *发件人：*Wanqiang Ji <[email protected]> > *发送时间：*2019-05-17 14:28 > *主题：*Re: Why yarn job was killed > *收件人：*"lk_hadoop"<[email protected]> > *抄送：*"user"<[email protected]> > > I think you'd better give us some logs about '1533044360609_6754589' in > RM. > > On Fri, May 17, 2019 at 11:25 AM lk_hadoop <[email protected]> wrote: > >> I can't find error message from logs. I want to know the reason,full log >> please see the attachments. >> >> 2019-05-17 >> ------------------------------ >> lk_hadoop >> ------------------------------ >> *发件人：*"lk_hadoop"<[email protected]> >> *发送时间：*2019-05-17 11:02 >> *主题：*Kylin Hive Column Cardinality Job killed by user. >> *收件人：*"user"<[email protected]> >> *抄送：* >> >> hi,all: >> I'm trying to load a hive table and calculate it's Column Cardinality >> ，but I found the MR job was killed by yarn : >> >> Message from yarn web : >> >> Aggregate Resource Allocation: 731002 MB-seconds, 190 vcore-seconds >> >> Logs about the ApplicationMaster syslog : >> >> 2019-05-17 10:21:43,167 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for >> application appattempt_1533044360609_6754589_000001 >> 2019-05-17 10:21:43,614 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: >> 2019-05-17 10:21:43,614 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, >> Service: , Ident: >> (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@1a75e76a) >> 2019-05-17 10:21:43,874 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. >> 2019-05-17 10:21:43,876 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in >> config null >> 2019-05-17 10:21:43,994 INFO [main] >> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output >> Committer Algorithm version is 1 >> 2019-05-17 10:21:43,994 INFO [main] >> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: >> FileOutputCommitter skip cleanup _temporary folders under output >> directory:false, ignore cleanup failures: false >> 2019-05-17 10:21:45,201 WARN [main] org.apache.hadoop.util.NativeCodeLoader: >> Unable to load native-hadoop library for your platform... using builtin-java >> classes where applicable >> 2019-05-17 10:21:45,256 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is >> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter >> 2019-05-17 10:21:45,647 INFO [main] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.mapreduce.jobhistory.EventType for class >> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler >> 2019-05-17 10:21:45,648 INFO [main] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher >> 2019-05-17 10:21:45,651 INFO [main] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher >> 2019-05-17 10:21:45,653 INFO [main] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher >> 2019-05-17 10:21:45,653 INFO [main] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class >> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler >> 2019-05-17 10:21:45,657 INFO [main] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher >> 2019-05-17 10:21:45,658 INFO [main] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter >> 2019-05-17 10:21:45,659 INFO [main] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for >> class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter >> 2019-05-17 10:21:45,736 INFO [main] >> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file >> system [hdfs://nameservice1:8020] >> 2019-05-17 10:21:45,811 INFO [main] >> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file >> system [hdfs://nameservice1:8020] >> 2019-05-17 10:21:45,874 INFO [main] >> org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file >> system [hdfs://nameservice1:8020] >> 2019-05-17 10:21:45,914 INFO [main] >> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job >> history data to the timeline server is not enabled >> 2019-05-17 10:21:45,973 INFO [main] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class >> org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler >> 2019-05-17 10:21:46,392 INFO [main] >> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from >> hadoop-metrics2.properties >> 2019-05-17 10:21:46,543 INFO [main] >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period >> at 10 second(s). >> 2019-05-17 10:21:46,543 INFO [main] >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics >> system started >> 2019-05-17 10:21:46,569 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for >> job_1533044360609_6754589 to jobTokenSecretManager >> 2019-05-17 10:21:47,160 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing >> job_1533044360609_6754589 because: not enabled; too many maps; too much >> input; too much RAM; >> 2019-05-17 10:21:47,340 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job >> job_1533044360609_6754589 = 32845338598. Number of splits = 14000 >> 2019-05-17 10:21:47,383 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for >> job job_1533044360609_6754589 = 1 >> 2019-05-17 10:21:47,383 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: >> job_1533044360609_6754589Job Transitioned from NEW to INITED >> 2019-05-17 10:21:47,384 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching >> normal, non-uberized, multi-container job job_1533044360609_6754589. >> 2019-05-17 10:21:47,495 INFO [main] org.apache.hadoop.ipc.CallQueueManager: >> Using callQueue: class java.util.concurrent.LinkedBlockingQueue >> queueCapacity: 100 >> 2019-05-17 10:21:47,530 INFO [Socket Reader #1 for port 35208] >> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 35208 >> 2019-05-17 10:21:47,540 INFO [Socket Reader #2 for port 35208] >> org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 35208 >> 2019-05-17 10:21:47,555 INFO [Socket Reader #3 for port 35208] >> org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 35208 >> 2019-05-17 10:21:47,592 INFO [Socket Reader #4 for port 35208] >> org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 35208 >> 2019-05-17 10:21:47,628 INFO [Socket Reader #5 for port 35208] >> org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 35208 >> 2019-05-17 10:21:47,709 INFO [main] >> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding >> protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server >> 2019-05-17 10:21:47,841 INFO [IPC Server Responder] >> org.apache.hadoop.ipc.Server: IPC Server Responder: starting >> 2019-05-17 10:21:47,856 INFO [IPC Server listener on 35208] >> org.apache.hadoop.ipc.Server: IPC Server listener on 35208: starting >> 2019-05-17 10:21:47,875 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated >> MRClientService at bdp-hdfs-10-10-8-33-39/10.8.33.39:35208 >> 2019-05-17 10:21:48,091 INFO [main] org.mortbay.log: Logging to >> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >> org.mortbay.log.Slf4jLog >> 2019-05-17 10:21:48,102 INFO [main] >> org.apache.hadoop.security.authentication.server.AuthenticationFilter: >> Unable to initialize FileSignerSecretProvider, falling back to use random >> secrets. >> 2019-05-17 10:21:48,124 INFO [main] org.apache.hadoop.http.HttpRequestLog: >> Http request log for http.requests.mapreduce is not defined >> 2019-05-17 10:21:48,142 INFO [main] org.apache.hadoop.http.HttpServer2: >> Added global filter 'safety' >> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) >> 2019-05-17 10:21:48,249 INFO [main] org.apache.hadoop.http.HttpServer2: >> Added filter AM_PROXY_FILTER >> (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to >> context mapreduce >> 2019-05-17 10:21:48,250 INFO [main] org.apache.hadoop.http.HttpServer2: >> Added filter AM_PROXY_FILTER >> (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to >> context static >> 2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: >> adding path spec: /mapreduce/* >> 2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: >> adding path spec: /ws/* >> 2019-05-17 10:21:48,285 INFO [main] org.apache.hadoop.http.HttpServer2: >> Jetty bound to port 35859 >> 2019-05-17 10:21:48,285 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4 >> 2019-05-17 10:21:48,377 INFO [main] org.mortbay.log: Extract >> jar:file:/data/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/hadoop-yarn-common-2.6.0-cdh5.14.0.jar!/webapps/mapreduce >> to /tmp/Jetty_0_0_0_0_35859_mapreduce____fnjnpb/webapp >> 2019-05-17 10:21:49,050 INFO [main] org.mortbay.log: Started >> [email protected]:35859 >> 2019-05-17 10:21:49,050 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: >> Web app /mapreduce started at 35859 >> 2019-05-17 10:21:49,604 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: >> Registered webapp guice modules >> 2019-05-17 10:21:49,639 INFO [main] org.apache.hadoop.ipc.CallQueueManager: >> Using callQueue: class java.util.concurrent.LinkedBlockingQueue >> queueCapacity: 3000 >> 2019-05-17 10:21:49,653 INFO [Socket Reader #1 for port 40180] >> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 40180 >> 2019-05-17 10:21:49,666 INFO [Socket Reader #3 for port 40180] >> org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 40180 >> 2019-05-17 10:21:49,667 INFO [Socket Reader #4 for port 40180] >> org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 40180 >> 2019-05-17 10:21:49,667 INFO [Socket Reader #5 for port 40180] >> org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 40180 >> 2019-05-17 10:21:49,702 INFO [Socket Reader #2 for port 40180] >> org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 40180 >> 2019-05-17 10:21:49,773 INFO [IPC Server Responder] >> org.apache.hadoop.ipc.Server: IPC Server Responder: starting >> 2019-05-17 10:21:49,793 INFO [IPC Server listener on 40180] >> org.apache.hadoop.ipc.Server: IPC Server listener on 40180: starting >> 2019-05-17 10:21:50,064 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: >> nodeBlacklistingEnabled:true >> 2019-05-17 10:21:50,065 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: >> maxTaskFailuresPerNode is 3 >> 2019-05-17 10:21:50,065 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: >> blacklistDisablePercent is 33 >> 2019-05-17 10:21:50,241 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: >> maxContainerCapability: <memory:20480, vCores:42> >> 2019-05-17 10:21:50,242 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: >> root.BASE_DATA >> 2019-05-17 10:21:50,246 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper >> limit on the thread pool size is 500 >> 2019-05-17 10:21:50,246 INFO [main] >> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The >> thread pool initial size is 10 >> 2019-05-17 10:21:50,297 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: >> job_1533044360609_6754589Job Transitioned from INITED to SETUP >> 2019-05-17 10:21:50,300 INFO [CommitterEvent Processor #0] >> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing >> the event EventType: JOB_SETUP >> 2019-05-17 10:21:50,657 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 1000 >> 2019-05-17 10:21:50,666 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 2000 >> 2019-05-17 10:21:50,667 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 3000 >> 2019-05-17 10:21:50,668 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 4000 >> 2019-05-17 10:21:50,670 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 5000 >> 2019-05-17 10:21:50,671 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 6000 >> 2019-05-17 10:21:50,672 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 7000 >> 2019-05-17 10:21:50,673 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 8000 >> 2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 9000 >> 2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 10000 >> 2019-05-17 10:21:50,675 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 11000 >> 2019-05-17 10:21:50,676 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 12000 >> 2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 13000 >> 2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 14000 >> >> .... >> >> 2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: >> task_1533044360609_6754589_m_013998 Task Transitioned from NEW to SCHEDULED >> 2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 28000 >> 2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: >> task_1533044360609_6754589_m_013999 Task Transitioned from NEW to SCHEDULED >> 2019-05-17 10:21:53,441 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: >> task_1533044360609_6754589_r_000000 Task Transitioned from NEW to SCHEDULED >> 2019-05-17 10:21:53,460 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: >> attempt_1533044360609_6754589_m_000000_0 TaskAttempt Transitioned from NEW >> to UNASSIGNED >> >> .... >> >> 2019-05-17 10:21:54,532 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: >> attempt_1533044360609_6754589_r_000000_0 TaskAttempt Transitioned from NEW >> to UNASSIGNED >> 2019-05-17 10:21:54,544 INFO [Thread-52] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: >> mapResourceRequest:<memory:4096, vCores:1> >> 2019-05-17 10:21:54,547 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 1000 >> 2019-05-17 10:21:54,551 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 2000 >> 2019-05-17 10:21:54,555 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 3000 >> 2019-05-17 10:21:54,559 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 4000 >> 2019-05-17 10:21:54,584 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 5000 >> 2019-05-17 10:21:54,587 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 6000 >> 2019-05-17 10:21:54,590 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 7000 >> 2019-05-17 10:21:54,648 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 8000 >> 2019-05-17 10:21:54,651 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 9000 >> 2019-05-17 10:21:54,653 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 10000 >> 2019-05-17 10:21:54,666 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 11000 >> 2019-05-17 10:21:54,668 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 12000 >> 2019-05-17 10:21:54,671 INFO [AsyncDispatcher event handler] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of >> event-queue in RMContainerAllocator is 13000 >> 2019-05-17 10:21:54,903 INFO [Thread-52] >> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: >> reduceResourceRequest:<memory:5120, vCores:1> >> >> .... >> >> >> >> I didn't find error log, full syslog please see attachments. >> >> >> 2019-05-17 >> ------------------------------ >> lk_hadoop >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] > >
