Repository: spark Updated Branches: refs/heads/master 07fd94e0d -> d4f10cbbe
[SPARK-20343][BUILD] Force Avro 1.7.7 in sbt build to resolve build failure in SBT Hadoop 2.6 master on Jenkins ## What changes were proposed in this pull request? This PR proposes to force Avro's version to 1.7.7 in core to resolve the build failure as below: ``` [error] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData [error] writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema)) [error] ``` https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/2770/consoleFull Note that this is a hack and should be removed in the future. ## How was this patch tested? I only tested this actually overrides the dependency. I tried many ways but I was unable to reproduce this in my local. Sean also tried the way I did but he was also unable to reproduce this. Please refer the comments in https://github.com/apache/spark/pull/17477#issuecomment-294094092 Author: hyukjinkwon <[email protected]> Closes #17651 from HyukjinKwon/SPARK-20343-sbt. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d4f10cbb Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d4f10cbb Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d4f10cbb Branch: refs/heads/master Commit: d4f10cbbe1b9d13e43d80a50d204781e1c5c2da9 Parents: 07fd94e Author: hyukjinkwon <[email protected]> Authored: Tue Apr 18 11:05:00 2017 +0100 Committer: Sean Owen <[email protected]> Committed: Tue Apr 18 11:05:00 2017 +0100 ---------------------------------------------------------------------- pom.xml | 1 + project/SparkBuild.scala | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/d4f10cbb/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index c117459..14370d9 100644 --- a/pom.xml +++ b/pom.xml @@ -142,6 +142,7 @@ <ivy.version>2.4.0</ivy.version> <oro.version>2.0.8</oro.version> <codahale.metrics.version>3.1.2</codahale.metrics.version> + <!-- Keep consistent with Avro vesion in SBT build for SPARK-20343 --> <avro.version>1.7.7</avro.version> <avro.mapred.classifier>hadoop2</avro.mapred.classifier> <jets3t.version>0.9.3</jets3t.version> http://git-wip-us.apache.org/repos/asf/spark/blob/d4f10cbb/project/SparkBuild.scala ---------------------------------------------------------------------- diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala index e52baf5..77dae28 100644 --- a/project/SparkBuild.scala +++ b/project/SparkBuild.scala @@ -318,8 +318,8 @@ object SparkBuild extends PomBuild { enable(MimaBuild.mimaSettings(sparkHome, x))(x) } - /* Generate and pick the spark build info from extra-resources */ - enable(Core.settings)(core) + /* Generate and pick the spark build info from extra-resources and override a dependency */ + enable(Core.settings ++ CoreDependencyOverrides.settings)(core) /* Unsafe settings */ enable(Unsafe.settings)(unsafe) @@ -444,6 +444,16 @@ object DockerIntegrationTests { } /** + * Overrides to work around sbt's dependency resolution being different from Maven's in Unidoc. + * + * Note that, this is a hack that should be removed in the future. See SPARK-20343 + */ +object CoreDependencyOverrides { + lazy val settings = Seq( + dependencyOverrides += "org.apache.avro" % "avro" % "1.7.7") +} + +/** * Overrides to work around sbt's dependency resolution being different from Maven's. */ object DependencyOverrides { --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
