Repository: spark Updated Branches: refs/heads/branch-2.0 684167862 -> c7e013f18
[SPARK-15456][PYSPARK] Fixed PySpark shell context initialization when HiveConf not present ## What changes were proposed in this pull request? When PySpark shell cannot find HiveConf, it will fallback to create a SparkSession from a SparkContext. This fixes a bug caused by using a variable to SparkContext before it was initialized. ## How was this patch tested? Manually starting PySpark shell and using the SparkContext Author: Bryan Cutler <[email protected]> Closes #13237 from BryanCutler/pyspark-shell-session-context-SPARK-15456. (cherry picked from commit 021c19702c720b4466b016498917d47f99000e13) Signed-off-by: Andrew Or <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c7e013f1 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c7e013f1 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c7e013f1 Branch: refs/heads/branch-2.0 Commit: c7e013f1888f084d6a02d6dd905a77c4f20d1db4 Parents: 6841678 Author: Bryan Cutler <[email protected]> Authored: Fri May 20 16:41:57 2016 -0700 Committer: Andrew Or <[email protected]> Committed: Fri May 20 16:42:09 2016 -0700 ---------------------------------------------------------------------- python/pyspark/shell.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/c7e013f1/python/pyspark/shell.py ---------------------------------------------------------------------- diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py index ef46d30..ac5ce87 100644 --- a/python/pyspark/shell.py +++ b/python/pyspark/shell.py @@ -44,9 +44,9 @@ try: .enableHiveSupport()\ .getOrCreate() except py4j.protocol.Py4JError: - spark = SparkSession(sc) + spark = SparkSession.builder.getOrCreate() except TypeError: - spark = SparkSession(sc) + spark = SparkSession.builder.getOrCreate() sc = spark.sparkContext atexit.register(lambda: sc.stop()) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
