yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds is your friend. Set it to 3600 and you will see logs of running apps being aggregated every hour.
-Gour From: Soheil Pourbafrani <[email protected]> Date: Friday, April 27, 2018 at 12:56 AM To: "[email protected]" <[email protected]> Subject: Yarn didn't read logs completely while app is running Using Hadoop 2.8.3, I set the following properties for log management, in yarn-site.xml: <!-- YARN LOG SETTINGS --> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>604800</value> </property> <property> <name>yarn.log-aggregation.retain-check-interval-seconds</name> <value>86400</value> </property> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/data/hadoop/yarn/log/app-logs</value> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value>/data/hadoop/yarn/log,/tmp/hadoop/yarn/log</value> </property> <property> <name>yarn.log.server.url</name> <value>http://snamenode:19888/jobhistory/logs</value> </property> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>10800</value> </property> <!-- END --> The Yarn behavior is: while an app is running logs writes in containers' local filesystem and after app finished, all logs from containers will roll into HDFS. The problem is when apps are running I can see the data apps prints and it just contains the log of running apps. After rolling logs in HDFS, the app printed data are in log files. How can I see all logs while an app is running?
