[
https://issues.apache.org/jira/browse/PIO-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930020#comment-15930020
]
ASF GitHub Bot commented on PIO-30:
-----------------------------------
GitHub user shimamoto opened a pull request:
https://github.com/apache/incubator-predictionio/pull/364
[PIO-30] Spark 2 support on sbt way
The goal of this PR is:
- Define the versions of Scala to build against in the crossScalaVersions
setting
- Scala 2.10 and 2.11
- At the project, run `make-distribution.sh`. Then, create all versions gzs
- PredictionIO_2.10-0.11.0-$VERSION.tar.gz <- This is intended for
Spark 1 users
- PredictionIO_2.11-0.11.0-$VERSION.tar.gz <- This is intended for
Spark 2 users
- SQLContext maintain the status quo
- I wish support for Scala 2.10 is deprecated as of next release
- Versions of Hadoop are 2.6+, which has something to do with storage hdfs
- 2.6 is in lib/spark(default)
- 2.7 is in lib/extra
@chanlee514 @dszeto
I did #345.
## TODO
Currently, only the traditional version of the test works. It needs to be
added new version (Spark 2) tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shimamoto/incubator-predictionio scala211
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-predictionio/pull/364.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #364
----
commit 432e0b85001bf4cd6f4d09e3a84dc95b2b5c7464
Author: shimamoto <[email protected]>
Date: 2017-03-09T12:07:14Z
Build against Scala 2.10 and 2.11.
commit 8d74bb8cc40619979b7af984ff4918759b6edba1
Author: shimamoto <[email protected]>
Date: 2017-03-17T05:50:43Z
Merge remote-tracking branch 'upstreamjp/develop' into scala211
commit 6a2dbdf865b271b1db0abf5dcebcdfe372d52b44
Author: shimamoto <[email protected]>
Date: 2017-03-17T07:56:03Z
fixup
----
> Cross build for different versions of scala and spark
> -----------------------------------------------------
>
> Key: PIO-30
> URL: https://issues.apache.org/jira/browse/PIO-30
> Project: PredictionIO
> Issue Type: Improvement
> Reporter: Marcin Ziemiński
> Assignee: Chan
> Fix For: 0.11.0
>
>
> The present version of Scala is 2.10 and Spark is 1.4, which is quite old.
> With Spark 2.0.0 come many performance improvements and features, that people
> will definitely like to add to their templates. I am also aware that past
> cannot be ignored and simply dumping 1.x might not be an option for other
> users.
> I propose setting up a crossbuild in sbt to build with scala 2.10 and Spark
> 1.6 and a separate one for Scala 2.11 and Spark 2.0. Most of the files will
> be consistent between versions including API. The problematic ones will be
> divided between additional source directories: src/main/scala-2.10/ and
> src/main/scala-2.11/. The dockerized tests should also take the two versions
> into consideration
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
