Hi guys : I want to start automating the output of counter stats, cluster size, etc... at the end of the main map reduce jobs which we run. Is there a simple way to do this ?
Here is my current thought : 1) Run all jobs from a driver class (we already do this). 2) At the end of each job, intercept the global counters and write them out to a text file. This would presumably be on the local fs. 3) Export the local filesystem. 4) Maybe the NameNode also has access to such data , maybe via an API (clearly, the hadoop web ui gets this data from somewhere, re in the "cluster summary" header.. -- Jay Vyas MMSB/UCHC
