Repository: spark
Updated Branches:
refs/heads/master 856928979 -> ed3aac791
[SPARK-5470][Core]use defaultClassLoader to load classes in KryoSerializer
Now KryoSerializer load classes of classesToRegister at the time of its
initialization. when we set spark.kryo.classesToRegister=class1, it will throw
SparkException("Failed to load class to register with Kryo".
because in KryoSerializer's initialization, classLoader cannot include class of
user's jars.
we need to use defaultClassLoader of Serializer in newKryo(), because executor
will reset defaultClassLoader of Serializer after Serializer's initialization.
thank zzcclp for reporting it to me.
Author: lianhuiwang <[email protected]>
Closes #4258 from lianhuiwang/SPARK-5470 and squashes the following commits:
73b719f [lianhuiwang] do the splitting and filtering during initialization
64cf306 [lianhuiwang] use defaultClassLoader to load classes of
classesToRegister in KryoSerializer
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ed3aac79
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ed3aac79
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ed3aac79
Branch: refs/heads/master
Commit: ed3aac791232560d2cfe25e3304e48873d48b338
Parents: 8569289
Author: lianhuiwang <[email protected]>
Authored: Fri Feb 6 11:00:35 2015 +0000
Committer: Sean Owen <[email protected]>
Committed: Fri Feb 6 11:00:35 2015 +0000
----------------------------------------------------------------------
.../org/apache/spark/serializer/KryoSerializer.scala | 11 ++---------
1 file changed, 2 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/ed3aac79/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
----------------------------------------------------------------------
diff --git
a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
index d56e23c..02158aa 100644
--- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
@@ -58,14 +58,6 @@ class KryoSerializer(conf: SparkConf)
private val classesToRegister = conf.get("spark.kryo.classesToRegister", "")
.split(',')
.filter(!_.isEmpty)
- .map { className =>
- try {
- Class.forName(className)
- } catch {
- case e: Exception =>
- throw new SparkException("Failed to load class to register with
Kryo", e)
- }
- }
def newKryoOutput() = new KryoOutput(bufferSize, math.max(bufferSize,
maxBufferSize))
@@ -97,7 +89,8 @@ class KryoSerializer(conf: SparkConf)
// Use the default classloader when calling the user registrator.
Thread.currentThread.setContextClassLoader(classLoader)
// Register classes given through spark.kryo.classesToRegister.
- classesToRegister.foreach { clazz => kryo.register(clazz) }
+ classesToRegister
+ .foreach { className => kryo.register(Class.forName(className, true,
classLoader)) }
// Allow the user to register their own classes by setting
spark.kryo.registrator.
userRegistrator
.map(Class.forName(_, true,
classLoader).newInstance().asInstanceOf[KryoRegistrator])
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
