Repository: spark Updated Branches: refs/heads/branch-1.2 db83acb1f -> 7a4be0b45
[YARN][SPARK-4929] Bug fix: fix the yarn-client code to support HA Nowadays, yarn-client will exit directly when the HA change happens no matter how many times the am should retry. The reason may be that the default final status only considerred the sys.exit, and the yarn-client HA cann't benefit from this. So we should distinct the default final status between client and cluster, because the SUCCEEDED status may cause the HA failed in client mode and UNDEFINED may cause the error reporter in cluster when using sys.exit. Author: huangzhaowei <[email protected]> Closes #3771 from SaintBacchus/YarnHA and squashes the following commits: c02bfcc [huangzhaowei] Improve the comment of the funciton 'getDefaultFinalStatus' 0e69924 [huangzhaowei] Bug fix: fix the yarn-client code to support HA (cherry picked from commit 5fde66163fe460d6f64b145047f76cc4ee33601a) Signed-off-by: Thomas Graves <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7a4be0b4 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7a4be0b4 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7a4be0b4 Branch: refs/heads/branch-1.2 Commit: 7a4be0b45f003ce92031d36bf74a736a87889026 Parents: db83acb Author: huangzhaowei <[email protected]> Authored: Wed Jan 7 08:10:42 2015 -0600 Committer: Thomas Graves <[email protected]> Committed: Wed Jan 7 08:11:14 2015 -0600 ---------------------------------------------------------------------- .../spark/deploy/yarn/ApplicationMaster.scala | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/7a4be0b4/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---------------------------------------------------------------------- diff --git a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala index 987b337..166e84e 100644 --- a/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala +++ b/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala @@ -60,7 +60,7 @@ private[spark] class ApplicationMaster(args: ApplicationMasterArguments, @volatile private var exitCode = 0 @volatile private var unregistered = false @volatile private var finished = false - @volatile private var finalStatus = FinalApplicationStatus.SUCCEEDED + @volatile private var finalStatus = getDefaultFinalStatus @volatile private var finalMsg: String = "" @volatile private var userClassThread: Thread = _ @@ -153,6 +153,20 @@ private[spark] class ApplicationMaster(args: ApplicationMasterArguments, } /** + * Set the default final application status for client mode to UNDEFINED to handle + * if YARN HA restarts the application so that it properly retries. Set the final + * status to SUCCEEDED in cluster mode to handle if the user calls System.exit + * from the application code. + */ + final def getDefaultFinalStatus() = { + if (isDriver) { + FinalApplicationStatus.SUCCEEDED + } else { + FinalApplicationStatus.UNDEFINED + } + } + + /** * unregister is used to completely unregister the application from the ResourceManager. * This means the ResourceManager will not retry the application attempt on your behalf if * a failure occurred. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
