It used the local mode of operation: org.apache.hadoop.mapred.LocalJobRunner
A JobTracker (via MiniMRCluster) is only required for simulating distributed tests. On Wed, Aug 8, 2012 at 2:27 AM, Mohit Anchlia <[email protected]> wrote: > I just wrote a test where fs.default.name is file:/// and > mapred.job.tracker is set to local. The test ran fine, I also see mapper > and reducer were invoked but what I am trying to understand is that how did > this run without specifying the job tracker port and which port task > tracker connected with job tracker. It's not clear from the output: > > Also what's the difference between this and bringing up miniDFS cluster? > > INFO org.apache.hadoop.mapred.FileInputFormat [main]: Total input paths to > proc > ess : 1 > INFO org.apache.hadoop.mapred.JobClient [main]: Running job: job_local_0001 > INFO org.apache.hadoop.mapred.Task [Thread-11]: Using > ResourceCalculatorPlugin > : null > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: numReduceTasks: 1 > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: io.sort.mb = 100 > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: data buffer = > 79691776/99614 > 720 > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: record buffer = > 262144/32768 > 0 > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z > ip 92127 > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z > ip 1 > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z > ip 92127 > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z > ip 1 > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: Starting flush of map > output > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: Finished spill 0 > INFO org.apache.hadoop.mapred.Task [Thread-11]: > Task:attempt_local_0001_m_00000 > 0_0 is done. And is in the process of commiting > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: > file:/c:/upb/dp/manch > lia-dp/depot/services/data-platform/trunk/analytics/geoinput/geo.dat:0+18 > INFO org.apache.hadoop.mapred.Task [Thread-11]: Task > 'attempt_local_0001_m_0000 > 00_0' done. > INFO org.apache.hadoop.mapred.Task [Thread-11]: Using > ResourceCalculatorPlugin > : null > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: > INFO org.apache.hadoop.mapred.Merger [Thread-11]: Merging 1 sorted segments > INFO org.apache.hadoop.mapred.Merger [Thread-11]: Down to the last > merge-pass, > with 1 segments left of total size: 26 bytes > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: I > nside reduce > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: O > utside reduce > INFO org.apache.hadoop.mapred.Task [Thread-11]: > Task:attempt_local_0001_r_00000 > 0_0 is done. And is in the process of commiting > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: > INFO org.apache.hadoop.mapred.Task [Thread-11]: Task > attempt_local_0001_r_00000 > 0_0 is allowed to commit now > INFO org.apache.hadoop.mapred.FileOutputCommitter [Thread-11]: Saved > output of > task 'attempt_local_0001_r_000000_0' to > file:/c:/upb/dp/manchlia-dp/depot/servic > es/data-platform/trunk/analytics/geooutput > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: reduce > reduce > INFO org.apache.hadoop.mapred.Task [Thread-11]: Task > 'attempt_local_0001_r_0000 > 00_0' done. > INFO org.apache.hadoop.mapred.JobClient [main]: map 100% reduce 100% > INFO org.apache.hadoop.mapred.JobClient [main]: Job complete: > job_local_0001 > INFO org.apache.hadoop.mapred.JobClient [main]: Counters: 15 > INFO org.apache.hadoop.mapred.JobClient [main]: FileSystemCounters > INFO org.apache.hadoop.mapred.JobClient [main]: FILE_BYTES_READ=458 > INFO org.apache.hadoop.mapred.JobClient [main]: > FILE_BYTES_WRITTEN=96110 > INFO org.apache.hadoop.mapred.JobClient [main]: Map-Reduce Framework > INFO org.apache.hadoop.mapred.JobClient [main]: Map input records=2 > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce shuffle bytes=0 > INFO org.apache.hadoop.mapred.JobClient [main]: Spilled Records=4 > INFO org.apache.hadoop.mapred.JobClient [main]: Map output bytes=20 > INFO org.apache.hadoop.mapred.JobClient [main]: Total committed heap > usage > (bytes)=321527808 > INFO org.apache.hadoop.mapred.JobClient [main]: Map input bytes=18 > INFO org.apache.hadoop.mapred.JobClient [main]: SPLIT_RAW_BYTES=142 > INFO org.apache.hadoop.mapred.JobClient [main]: Combine input records=0 > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce input records=2 > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce input groups=1 > INFO org.apache.hadoop.mapred.JobClient [main]: Combine output > records=0 > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce output records=1 > INFO org.apache.hadoop.mapred.JobClient [main]: Map output records=2 > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [main]: Inside > reduce > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [main]: Outsid > e reduce > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.547 sec > Results : > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 -- Harsh J
