Sorry about this late response, but Apache Rumen( http://hadoop.apache.org/docs/mapreduce/r0.22.0/rumen.html) is a good solution to extract several informations from job log.
-- Thiago Vieira On Mon, Aug 27, 2012 at 10:54 AM, Raj Vishwanathan <[email protected]>wrote: > You can extract the shuffle time from the job log. > > Take a look at > > https://github.com/rajvish/hadoop-summary > > > Raj > > > > >________________________________ > > From: Bertrand Dechoux <[email protected]> > >To: [email protected] > >Sent: Monday, August 27, 2012 12:57 AM > >Subject: Re: Measuring Shuffle time for MR job > > > >Shuffle time is considered as part of the reduce step. Without reduce, > >there is no need for shuffling. > >One way to measure it would be using the full reduce time with a > >'/dev/null' reducer. > > > >I am not aware of any way to measure it. > > > >Regards > > > >Bertrand > > > >On Mon, Aug 27, 2012 at 8:18 AM, praveenesh kumar <[email protected] > >wrote: > > > >> Is there a way to know the total shuffle time of a map-reduce job - I > mean > >> some command or output that can tell that ? > >> > >> I want to measure total map, total shuffle and total reduce time for my > MR > >> job -- how can I achieve it ? I am using hadoop 0.20.205 > >> > >> > >> Regards, > >> Praveenesh > >> > > > > > > > >-- > >Bertrand Dechoux > > > > > > >
