Repository: spark Updated Branches: refs/heads/branch-1.4 85cab3482 -> 959c7b6ca
[SPARK-7451] [YARN] Preemption of executors is counted as failure causing Spark job to fail Added a check to handle container exit status for the preemption scenario, log an INFO message in such cases and move on. andrewor14 Author: Ashwin Shankar <[email protected]> Closes #5993 from ashwinshankar77/SPARK-7451 and squashes the following commits: 90900cf [Ashwin Shankar] Fix log info message cf8b6cf [Ashwin Shankar] Stop counting preemption of executors as failure (cherry picked from commit b6c797b08cbd08d7aab59ad0106af0f5f41ef186) Signed-off-by: Sandy Ryza <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/959c7b6c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/959c7b6c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/959c7b6c Branch: refs/heads/branch-1.4 Commit: 959c7b6ca422082de9e89703e3c7b926d93d30d1 Parents: 85cab34 Author: Ashwin Shankar <[email protected]> Authored: Fri May 8 17:51:00 2015 -0700 Committer: Sandy Ryza <[email protected]> Committed: Fri May 8 17:51:46 2015 -0700 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/959c7b6c/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---------------------------------------------------------------------- diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala index b8f42da..88d68d5 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala @@ -373,7 +373,9 @@ private[yarn] class YarnAllocator( // Hadoop 2.2.X added a ContainerExitStatus we should switch to use // there are some exit status' we shouldn't necessarily count against us, but for // now I think its ok as none of the containers are expected to exit - if (completedContainer.getExitStatus == -103) { // vmem limit exceeded + if (completedContainer.getExitStatus == ContainerExitStatus.PREEMPTED) { + logInfo("Container preempted: " + containerId) + } else if (completedContainer.getExitStatus == -103) { // vmem limit exceeded logWarning(memLimitExceededLogMessage( completedContainer.getDiagnostics, VMEM_EXCEEDED_PATTERN)) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
