Repository: spark Updated Branches: refs/heads/branch-1.2 42b9d0d31 -> b026546e3
[SPARK-4381][Streaming]Add warning log when user set spark.master to local in Spark Streaming and there's no job executed Author: jerryshao <[email protected]> Closes #3244 from jerryshao/SPARK-4381 and squashes the following commits: d2486c7 [jerryshao] Improve the warning log d726e85 [jerryshao] Add local[1] to the filter condition eca428b [jerryshao] Add warning log (cherry picked from commit fef27b29431c2adadc17580f26c23afa6a3bd1d2) Signed-off-by: Tathagata Das <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b026546e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b026546e Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b026546e Branch: refs/heads/branch-1.2 Commit: b026546e3a2195a7e6106af3a5b7370cdb850052 Parents: 42b9d0d Author: jerryshao <[email protected]> Authored: Tue Nov 25 05:36:29 2014 -0800 Committer: Tathagata Das <[email protected]> Committed: Tue Nov 25 05:37:13 2014 -0800 ---------------------------------------------------------------------- .../scala/org/apache/spark/streaming/StreamingContext.scala | 5 +++++ 1 file changed, 5 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/b026546e/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---------------------------------------------------------------------- diff --git a/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala b/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala index ec59221..ecab551 100644 --- a/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala +++ b/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala @@ -121,6 +121,11 @@ class StreamingContext private[streaming] ( } } + if (sc.conf.get("spark.master") == "local" || sc.conf.get("spark.master") == "local[1]") { + logWarning("spark.master should be set as local[n], n > 1 in local mode if you have receivers" + + " to get data, otherwise Spark jobs will not get resources to process the received data.") + } + private[streaming] val conf = sc.conf private[streaming] val env = SparkEnv.get --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
