Using Hadoop 2.8.3, I set the following properties for log management, in
yarn-site.xml:
<!-- YARN LOG SETTINGS -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
<property>
<name>yarn.log-aggregation.retain-check-interval-seconds</name>
<value>86400</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/data/hadoop/yarn/log/app-logs</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/data/hadoop/yarn/log,/tmp/hadoop/yarn/log</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://snamenode:19888/jobhistory/logs</value>
</property>
<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>10800</value>
</property>
<!-- END -->
The Yarn behavior is: while an app is running logs writes in containers'
local filesystem and after app finished, all logs from containers will roll
into HDFS.
The problem is when apps are running I can see the data apps prints and it
just contains the log of running apps. After rolling logs in HDFS, the app
printed data are in log files.
How can I see all logs while an app is running?
