[
https://issues.apache.org/jira/browse/HADOOP-19835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18081099#comment-18081099
]
ASF GitHub Bot commented on HADOOP-19835:
-----------------------------------------
K0K0V0K commented on code in PR #8331:
URL: https://github.com/apache/hadoop/pull/8331#discussion_r3246813391
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:
##########
@@ -503,7 +503,9 @@ private List<String> setupAMCommand(Configuration jobConf) {
}
}
- vargs.add(MRJobConfig.APPLICATION_MASTER_CLASS);
+ String amClass = jobConf.get("yarn.app.mapreduce.am",
Review Comment:
Thanks @lewismc for the improvement!
For me, the proposal looks good overall.
Since we already have **mapreduce.job.map.class** and
**mapreduce.job.reduce.class**, I would suggest naming the new configuration
property **mapreduce.job.am.class** for consistency.
May I also ask which Java versions were tested with this change? I was
initially wondering whether the module system could cause any issues on newer
JDKs, but I think this approach should work fine.
FYI, while I am not fully aware of the intended deployment use case, I
believe there are at least two possible mechanisms for making the class
available on the NodeManagers:
* Bundling it directly with the application
* Adding it to the MapReduce framework tarball via
`org.apache.hadoop.mapred.uploader.FrameworkUploader`
Could I also ask you to add the new property here as well?
https://github.com/apache/hadoop/blob/22008a241a81a5f0a77cf55ead3546502c71df99/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java#L167
That would provide at least some limited control over what kinds of classes
can be loaded through this mechanism.
> Make MapReduce Application Master class configurable in YARNRunner
> ------------------------------------------------------------------
>
> Key: HADOOP-19835
> URL: https://issues.apache.org/jira/browse/HADOOP-19835
> Project: Hadoop Common
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 3.4.3
> Reporter: Lewis John McGibbney
> Assignee: Lewis John McGibbney
> Priority: Minor
> Labels: patch, pull-request-available
>
> YARNRunner currently always uses {{MRJobConfig.APPLICATION_MASTER_CLASS}}
> when building the command to start the Application Master, and does not read
> {{{}yarn.app.mapreduce.am{}}}. As a result, custom AM classes (e.g.
> Celeborn’s {{{}MRAppMasterWithCeleborn{}}}) are never used when submitting
> jobs via the standard job client.
> This patch changes YARNRunner to take the AM class from configuration: it
> uses {{jobConf.get("yarn.app.mapreduce.am",
> MRJobConfig.APPLICATION_MASTER_CLASS)}} and passes that value into the launch
> arguments. If {{yarn.app.mapreduce.am}} is set, that class is used; otherwise
> behavior is unchanged (default AM class). This allows remote shuffle and
> other custom AMs to be used without forking the job client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
