Yes, singular JVM (The test JVM itself) and the latter approach (no TT/JT daemons).
On Wed, Aug 8, 2012 at 4:50 AM, Mohit Anchlia <[email protected]> wrote: > On Tue, Aug 7, 2012 at 2:08 PM, Harsh J <[email protected]> wrote: > >> It used the local mode of operation: >> org.apache.hadoop.mapred.LocalJobRunner >> >> > In localmode everything is done inside the same JVM i.e. > tasktracker,jobtracker etc. all run in the same JVM. Or does it mean that > none of those processes run everything is pipelined in the same process on > the local file system. > > >> A JobTracker (via MiniMRCluster) is only required for simulating >> distributed tests. >> >> On Wed, Aug 8, 2012 at 2:27 AM, Mohit Anchlia <[email protected]> >> wrote: >> > I just wrote a test where fs.default.name is file:/// and >> > mapred.job.tracker is set to local. The test ran fine, I also see mapper >> > and reducer were invoked but what I am trying to understand is that how >> did >> > this run without specifying the job tracker port and which port task >> > tracker connected with job tracker. It's not clear from the output: >> > >> > Also what's the difference between this and bringing up miniDFS cluster? >> > >> > INFO org.apache.hadoop.mapred.FileInputFormat [main]: Total input paths >> to >> > proc >> > ess : 1 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Running job: >> job_local_0001 >> > INFO org.apache.hadoop.mapred.Task [Thread-11]: Using >> > ResourceCalculatorPlugin >> > : null >> > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: numReduceTasks: 1 >> > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: io.sort.mb = 100 >> > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: data buffer = >> > 79691776/99614 >> > 720 >> > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: record buffer = >> > 262144/32768 >> > 0 >> > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup >> [Thread-11]: z >> > ip 92127 >> > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup >> [Thread-11]: z >> > ip 1 >> > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup >> [Thread-11]: z >> > ip 92127 >> > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup >> [Thread-11]: z >> > ip 1 >> > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: Starting flush of map >> > output >> > INFO org.apache.hadoop.mapred.MapTask [Thread-11]: Finished spill 0 >> > INFO org.apache.hadoop.mapred.Task [Thread-11]: >> > Task:attempt_local_0001_m_00000 >> > 0_0 is done. And is in the process of commiting >> > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: >> > file:/c:/upb/dp/manch >> > lia-dp/depot/services/data-platform/trunk/analytics/geoinput/geo.dat:0+18 >> > INFO org.apache.hadoop.mapred.Task [Thread-11]: Task >> > 'attempt_local_0001_m_0000 >> > 00_0' done. >> > INFO org.apache.hadoop.mapred.Task [Thread-11]: Using >> > ResourceCalculatorPlugin >> > : null >> > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: >> > INFO org.apache.hadoop.mapred.Merger [Thread-11]: Merging 1 sorted >> segments >> > INFO org.apache.hadoop.mapred.Merger [Thread-11]: Down to the last >> > merge-pass, >> > with 1 segments left of total size: 26 bytes >> > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: >> > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup >> [Thread-11]: I >> > nside reduce >> > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup >> [Thread-11]: O >> > utside reduce >> > INFO org.apache.hadoop.mapred.Task [Thread-11]: >> > Task:attempt_local_0001_r_00000 >> > 0_0 is done. And is in the process of commiting >> > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: >> > INFO org.apache.hadoop.mapred.Task [Thread-11]: Task >> > attempt_local_0001_r_00000 >> > 0_0 is allowed to commit now >> > INFO org.apache.hadoop.mapred.FileOutputCommitter [Thread-11]: Saved >> > output of >> > task 'attempt_local_0001_r_000000_0' to >> > file:/c:/upb/dp/manchlia-dp/depot/servic >> > es/data-platform/trunk/analytics/geooutput >> > INFO org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: reduce > >> reduce >> > INFO org.apache.hadoop.mapred.Task [Thread-11]: Task >> > 'attempt_local_0001_r_0000 >> > 00_0' done. >> > INFO org.apache.hadoop.mapred.JobClient [main]: map 100% reduce 100% >> > INFO org.apache.hadoop.mapred.JobClient [main]: Job complete: >> > job_local_0001 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Counters: 15 >> > INFO org.apache.hadoop.mapred.JobClient [main]: FileSystemCounters >> > INFO org.apache.hadoop.mapred.JobClient [main]: FILE_BYTES_READ=458 >> > INFO org.apache.hadoop.mapred.JobClient [main]: >> > FILE_BYTES_WRITTEN=96110 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Map-Reduce Framework >> > INFO org.apache.hadoop.mapred.JobClient [main]: Map input records=2 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce shuffle >> bytes=0 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Spilled Records=4 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Map output bytes=20 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Total committed heap >> > usage >> > (bytes)=321527808 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Map input bytes=18 >> > INFO org.apache.hadoop.mapred.JobClient [main]: SPLIT_RAW_BYTES=142 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Combine input >> records=0 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce input >> records=2 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce input >> groups=1 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Combine output >> > records=0 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Reduce output >> records=1 >> > INFO org.apache.hadoop.mapred.JobClient [main]: Map output records=2 >> > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [main]: >> Inside >> > reduce >> > INFO com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [main]: >> Outsid >> > e reduce >> > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.547 sec >> > Results : >> > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 >> >> >> >> -- >> Harsh J >> -- Harsh J
