Thanks for the reply, As it turns out that didn't help, IO is used even more as each reducer is copying and sorting. What are the options? Is there an option to limit reduce - > copy and reduce - > sort somehow?
Thanks, Marek M. ________________________________ From: Mostafa Gaber [[email protected]] Sent: Monday, February 06, 2012 6:50 PM To: [email protected] Subject: Re: Reducer IO Hello Marek, I think you can increase number of reducers for your MR job so as to reduce the amount of intermediate key-value pairs assigned to each reducer. Note also that the number of reducers is dependent on your job and how the output should be produced. On Mon, Feb 6, 2012 at 11:37 AM, Marek Miglinski <[email protected]<mailto:[email protected]>> wrote: Hey, I have a mapreduce job (transactions loader) and the main problem of it is "reduce->copy" and "reduce->sort" phase which takes all IO and uses all disk resources, what are the possible ways to reduce this load? My cloud settings are: ioSortFactor=80 ioSortMb=800 (mapredChildJavaOpts=Xmx1152m) I can lower those settings, what else can I tweak? Thanks, Marek M. -- Best Regards, Mostafa Ead
