One nice feature is to attach different configuration for different job. You can do it either by command line "-conf" or programatically. We are using that extensively while experimenting different setting.
Zhu, Guojun Modeling Sr Graduate 571-3824370 [email protected] Financial Engineering Freddie Mac Radim Kolar <[email protected]> 05/10/2012 07:56 AM Please respond to [email protected] To [email protected] cc Subject Re: max 1 mapper per node > We've been against these 'features' since it leads to very bad > behaviour across the cluster with multiple apps/users etc. Its not new feature, its extension of existing resource scheduling which works good enough only for RAM. There are 2 other resources - CPU cores and network IO which needs to be considered. We have job which is doing lot of network IO in mapper and its desirable to run mappers on different nodes even if reading blocks from HDFS will not be local. Our second job is burning all CPU cores on machine while doing computations, its important for mappers not to land on same node.
