[
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kam Kasravi resolved GEARPUMP-185.
----------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.8.0)
0.8.2
> Yarn kills Apache Gearpump's worker if the memory usage is too high
> -------------------------------------------------------------------
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
> Issue Type: Bug
> Environment: yarn
> Reporter: Piotr Grabuszynski
> Assignee: Kam Kasravi
> Fix For: 0.8.2
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_000003 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
> |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
> |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/stderr
>
> |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
> ...
> |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors'
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280 5.3 3.1 5406960 261356 pts/13 Sl+ 09:28 0:10
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=master
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417 7.7 1.8 5392212 154728 pts/27 Sl+ 09:29 0:05
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true
> -DlogFilename=worker
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
> -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0 3.1 4572132 255776 pts/27 Sl+ 09:33 0:11
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761 132 5.2 7268592 427200 pts/27 Sl+ 09:33 2:31
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 [...]
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 ...
> {code}
> h3. Steps to Reproduce
> To generate trafic I've used such application:
> {code}
> kafka source
> / \
> kafka sink hbase sink
> {code}
> Where kafka source write to the same topic as kafka source reads.
> Send some message to topic.
> Worker will disappear after few minutes (1-40 min depends on how much memory
> worker has);
> h3. Actual Results
> Worker has been killed
> h3. Additional information
> Increasing value of env WORKERS_MEMORY_LIMIT in gearpump-broker doesn't work.
> It increases only running time.
> It might be a problem with some java properties (hip size or sth). It is also
> possible there's some memory lick f.e. in hbase sink.
> Bigger piece of YARN log:
> {code}
> 2016-07-20 09:04:22,217 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 27737 for container-id
> container_e05_1468841606343_0021_01_000002: 285.7 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:22,270 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2046 for container-id
> container_e05_1468841606343_0128_01_000001: 356.7 MB of 1 GB physical memory
> used; 2.5 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:22,317 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2159 for container-id
> container_e05_1468841606343_0128_01_000003: 2.0 GB of 2 GB physical memory
> used; 26.8 GB of 4.2 GB virtual memory used
> 2016-07-20 09:04:22,378 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2116 for container-id
> container_e05_1468841606343_0128_01_000002: 348.0 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,424 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 27737 for container-id
> container_e05_1468841606343_0021_01_000002: 285.7 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,494 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2046 for container-id
> container_e05_1468841606343_0128_01_000001: 356.7 MB of 1 GB physical memory
> used; 2.5 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,538 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2159 for container-id
> container_e05_1468841606343_0128_01_000003: 2.0 GB of 2 GB physical memory
> used; 26.8 GB of 4.2 GB virtual memory used
> 2016-07-20 09:04:25,539 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Process tree for container: container_e05_1468841606343_0128_01_000003 has
> processes older than 1 iteration running over the configured limit.
> Limit=2147483648, current usage = 2167832576
> 2016-07-20 09:04:25,547 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Container [pid=2159,containerID=container_e05_1468841606343_0128_01_000003]
> is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB
> physical memory used; 26.8 GB of 4.2 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_e05_1468841606343_0128_01_000003 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/lib/*
> -Dgearpump.cluster.masters.0=cdh-worker-2.node.envname.consul:54946
> -Dgearpump.log.daemon.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
>
> -Dgearpump.worker-resource-manager-container-id=container_e05_1468841606343_0128_01_000003
>
> -Dgearpump.home=/data1/hadoop/yarn/nm/usercache/cf/appcache/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/pack/gearpump-2.11-0.8.0
>
> -Dgearpump.log.application.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
> -Dgearpump.hostname=cdh-worker-2.node.envname.consul
> io.gearpump.cluster.main.Worker 2>&1 | /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/stderr
>
> |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/stderr
>
> |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-yarn/lib/*
> -Dgearpump.cluster.masters.0=cdh-worker-2.node.envname.consul:54946
> -Dgearpump.log.daemon.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
>
> -Dgearpump.worker-resource-manager-container-id=container_e05_1468841606343_0128_01_000003
>
> -Dgearpump.home=/data1/hadoop/yarn/nm/usercache/cf/appcache/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003/pack/gearpump-2.11-0.8.0
>
> -Dgearpump.log.application.dir=/var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
> -Dgearpump.hostname=cdh-worker-2.node.envname.consul
> io.gearpump.cluster.main.Worker
> |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump7648852439078424047.conf -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/
> |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1
> -Dgearpump.master.starttime=2016-07-20-08-36
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true
> -Dgearpump.config.file=/tmp/gearpump7311147404608349128.conf -cp
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/netty-3.8.0.Final.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-streaming_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-guava-16.0.1.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-metrics-graphite-3.0.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-reflect-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-log4j12-1.7.7.jar:pack/gearpump-2.11-0.8.0/lib/log4j-1.2.17.jar:pack/gearpump-2.11-0.8.0/lib/akka-slf4j_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-parser-combinators_2.11-1.0.4.jar:pack/gearpump-2.11-0.8.0/lib/akka-actor_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-java8-compat_2.11-0.7.0.jar:pack/gearpump-2.11-0.8.0/lib/slf4j-api-1.7.16.jar:pack/gearpump-2.11-0.8.0/lib/akka-agent_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-library-2.11.8.jar:pack/gearpump-2.11-0.8.0/lib/commons-lang-2.6.jar:pack/gearpump-2.11-0.8.0/lib/akka-remote_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/config-1.3.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-protobuf_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-core_2.11-0.8.0.jar:pack/gearpump-2.11-0.8.0/lib/akka-kernel_2.11-2.4.2.jar:pack/gearpump-2.11-0.8.0/lib/scala-stm_2.11-0.7.jar:pack/gearpump-2.11-0.8.0/lib/uncommons-maths-1.2.2a.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-gs-collections-6.2.0.jar:pack/gearpump-2.11-0.8.0/lib/gearpump-shaded-akka-kryo_2.11-0.3.2.jar:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scrooge_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-javadoc.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-generator.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-thrift.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-column.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-protobuf.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-scala_2.10.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-test-hadoop2.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-common.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-pig-bundle.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-annotations-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/parquet-format-sources.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-common-2.6.0-cdh5.5.1-tests.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/hadoop-nfs-2.6.0-cdh5.5.1.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib
> 2016-07-20 09:04:25,548 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_e05_1468841606343_0128_01_000003 transitioned from
> RUNNING to KILLING
> 2016-07-20 09:04:25,548 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Cleaning up container container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:25,548 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Removed ProcessTree with root 2159
> 2016-07-20 09:04:25,584 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2116 for container-id
> container_e05_1468841606343_0128_01_000002: 348.0 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:25,621 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code
> from container container_e05_1468841606343_0128_01_000003 is : 143
> 2016-07-20 09:04:25,655 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_e05_1468841606343_0128_01_000003 transitioned from
> KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
> 2016-07-20 09:04:25,655 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting
> absolute path :
> /data1/hadoop/yarn/nm/usercache/cf/appcache/application_1468841606343_0128/container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:25,656 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=cf
> OPERATION=Container Finished - Killed TARGET=ContainerImpl
> RESULT=SUCCESS APPID=application_1468841606343_0128
> CONTAINERID=container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:25,656 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_e05_1468841606343_0128_01_000003 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> 2016-07-20 09:04:25,656 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Removing container_e05_1468841606343_0128_01_000003 from application
> application_1468841606343_0128
> 2016-07-20 09:04:25,656 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
> Considering container container_e05_1468841606343_0128_01_000003 for
> log-aggregation
> 2016-07-20 09:04:25,656 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
> event CONTAINER_STOP for appId application_1468841606343_0128
> 2016-07-20 09:04:25,656 INFO
> org.apache.spark.network.yarn.YarnShuffleService: Stopping container
> container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:27,703 INFO
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed
> completed containers from NM context:
> [container_e05_1468841606343_0128_01_000003]
> 2016-07-20 09:04:28,584 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Stopping resource-monitoring for container_e05_1468841606343_0128_01_000003
> 2016-07-20 09:04:28,594 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 27737 for container-id
> container_e05_1468841606343_0021_01_000002: 285.7 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:28,603 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2046 for container-id
> container_e05_1468841606343_0128_01_000001: 356.7 MB of 1 GB physical memory
> used; 2.5 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:28,613 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 2116 for container-id
> container_e05_1468841606343_0128_01_000002: 348.1 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> 2016-07-20 09:04:31,623 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 27737 for container-id
> container_e05_1468841606343_0021_01_000002: 285.7 MB of 1 GB physical memory
> used; 2.4 GB of 2.1 GB virtual memory used
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
