[
https://issues.apache.org/jira/browse/PIO-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144973#comment-16144973
]
ASF GitHub Bot commented on PIO-116:
------------------------------------
GitHub user marevol opened a pull request:
https://github.com/apache/incubator-predictionio/pull/427
[PIO-116] PySpark Support
This PR provides PySpark support with minimum PIO changes.
1. Support pyspark on pio-shell
2. Add python files to use pyspark
3. Add --main-py-file option to "pio train" to submit .py file to spark
Note that this provides only fixes for Spark 2.x.
(because this fixes expect to use SparkML)
Sample project is:
https://github.com/jpioug/predictionio-template-iris
(For prediction API, Scala code is used.)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marevol/incubator-predictionio pyspark
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-predictionio/pull/427.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #427
----
commit ee28fcf139c6ac8184d990cbdc4d43b00ff483fd
Author: Shinsuke Sugaya <[email protected]>
Date: 2017-08-22T09:47:05Z
add pyspark sub-command
commit 97f0343691ff1ca98f1ce65fc8ad3e25df6cd15b
Author: Shinsuke Sugaya <[email protected]>
Date: 2017-08-27T14:16:18Z
replace with values.toString
commit 2970397a6024f17872011979edcae1712f8a4362
Author: Shinsuke Sugaya <[email protected]>
Date: 2017-08-28T10:04:24Z
add --main-py-file option to train
----
> PySpark Support
> ---------------
>
> Key: PIO-116
> URL: https://issues.apache.org/jira/browse/PIO-116
> Project: PredictionIO
> Issue Type: New Feature
> Components: Core
> Reporter: Shinsuke Sugaya
> Assignee: Shinsuke Sugaya
>
> This provides PySpark support with minimum PIO changes.
> 1. Support pyspark on pio-shell
> 2. Add python files to use pyspark
> 3. Add --main-py-file option to "pio train" to submit .py file to spark
> Note that this provides only fixes for Spark 2.x.
> (because this fixes expect to use SparkML)
> Sample project is:
> https://github.com/jpioug/predictionio-template-iris
> (For prediction API, Scala code is used.)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
