Repository: spark Updated Branches: refs/heads/branch-1.5 3c33931aa -> 03a8a889a
[SPARK-9705] [DOC] fix docs about Python version cc JoshRosen Author: Davies Liu <[email protected]> Closes #8245 from davies/python_doc. (cherry picked from commit de3223872a217c5224ba7136604f6b7753b29108) Signed-off-by: Reynold Xin <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/03a8a889 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/03a8a889 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/03a8a889 Branch: refs/heads/branch-1.5 Commit: 03a8a889a98ab30e4d33dc1a415aa84253111ffa Parents: 3c33931 Author: Davies Liu <[email protected]> Authored: Tue Aug 18 22:11:27 2015 -0700 Committer: Reynold Xin <[email protected]> Committed: Tue Aug 18 22:11:32 2015 -0700 ---------------------------------------------------------------------- docs/configuration.md | 6 +++++- docs/programming-guide.md | 12 ++++++++++-- 2 files changed, 15 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/03a8a889/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index 3214709..4a6e4dd 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1561,7 +1561,11 @@ The following variables can be set in `spark-env.sh`: </tr> <tr> <td><code>PYSPARK_PYTHON</code></td> - <td>Python binary executable to use for PySpark.</td> + <td>Python binary executable to use for PySpark in both driver and workers (default is `python`).</td> + </tr> + <tr> + <td><code>PYSPARK_DRIVER_PYTHON</code></td> + <td>Python binary executable to use for PySpark in driver only (default is PYSPARK_PYTHON).</td> </tr> <tr> <td><code>SPARK_LOCAL_IP</code></td> http://git-wip-us.apache.org/repos/asf/spark/blob/03a8a889/docs/programming-guide.md ---------------------------------------------------------------------- diff --git a/docs/programming-guide.md b/docs/programming-guide.md index ae712d6..982c5ea 100644 --- a/docs/programming-guide.md +++ b/docs/programming-guide.md @@ -85,8 +85,8 @@ import org.apache.spark.SparkConf <div data-lang="python" markdown="1"> -Spark {{site.SPARK_VERSION}} works with Python 2.6 or higher (but not Python 3). It uses the standard CPython interpreter, -so C libraries like NumPy can be used. +Spark {{site.SPARK_VERSION}} works with Python 2.6+ or Python 3.4+. It can use the standard CPython interpreter, +so C libraries like NumPy can be used. It also works with PyPy 2.3+. To run Spark applications in Python, use the `bin/spark-submit` script located in the Spark directory. This script will load Spark's Java/Scala libraries and allow you to submit applications to a cluster. @@ -104,6 +104,14 @@ Finally, you need to import some Spark classes into your program. Add the follow from pyspark import SparkContext, SparkConf {% endhighlight %} +PySpark requires the same minor version of Python in both driver and workers. It uses the default python version in PATH, +you can specify which version of Python you want to use by `PYSPARK_PYTHON`, for example: + +{% highlight bash %} +$ PYSPARK_PYTHON=python3.4 bin/pyspark +$ PYSPARK_PYTHON=/opt/pypy-2.5/bin/pypy bin/spark-submit examples/src/main/python/pi.py +{% endhighlight %} + </div> </div> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
