Repository: spark Updated Branches: refs/heads/branch-2.3 f2c058565 -> d24d13179
[SPARK-23422][CORE] YarnShuffleIntegrationSuite fix when SPARK_PREPEN⦠â¦D_CLASSES set to 1 ## What changes were proposed in this pull request? YarnShuffleIntegrationSuite fails when SPARK_PREPEND_CLASSES set to 1. Normally mllib built before yarn module. When SPARK_PREPEND_CLASSES used mllib classes are on yarn test classpath. Before 2.3 that did not cause issues. But 2.3 has SPARK-22450, which registered some mllib classes with the kryo serializer. Now it dies with the following error: ` 18/02/13 07:33:29 INFO SparkContext: Starting job: collect at YarnShuffleIntegrationSuite.scala:143 Exception in thread "dag-scheduler-event-loop" java.lang.NoClassDefFoundError: breeze/linalg/DenseMatrix ` In this PR NoClassDefFoundError caught only in case of testing and then do nothing. ## How was this patch tested? Automated: Pass the Jenkins. Author: Gabor Somogyi <[email protected]> Closes #20608 from gaborgsomogyi/SPARK-23422. (cherry picked from commit 44e20c42254bc6591b594f54cd94ced5fcfadae3) Signed-off-by: Marcelo Vanzin <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d24d1317 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d24d1317 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d24d1317 Branch: refs/heads/branch-2.3 Commit: d24d13179f0e9d125eaaebfcc225e1ec30c5cb83 Parents: f2c0585 Author: Gabor Somogyi <[email protected]> Authored: Thu Feb 15 03:52:40 2018 -0800 Committer: Marcelo Vanzin <[email protected]> Committed: Thu Feb 15 03:52:55 2018 -0800 ---------------------------------------------------------------------- .../src/main/scala/org/apache/spark/serializer/KryoSerializer.scala | 1 + 1 file changed, 1 insertion(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/d24d1317/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala index 538ae05..72427dd 100644 --- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala +++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala @@ -206,6 +206,7 @@ class KryoSerializer(conf: SparkConf) kryo.register(clazz) } catch { case NonFatal(_) => // do nothing + case _: NoClassDefFoundError if Utils.isTesting => // See SPARK-23422. } } --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
