wzx140 commented on issue #9738: URL: https://github.com/apache/iceberg/issues/9738#issuecomment-2106112085
@adevore3 Kryo has better serialization performance than JavaSerializer. Setting global serialization to JavaSerializer is not a good solution. From the stack, when deserializing SerializableByteBufferMap value, Kryo received int value that cannot be resolved to byteBuffer. Continuing to look at the org.apache.iceberg.SerializableByteBufferMap#writeReplace method, it can be observed that this Java serialization method will be serialized in the form of int[], byte[][]. So this exception may be related to inconsistent serialization and deserialization methods I have found a solution. See https://github.com/apache/iceberg/issues/446 ```java public class IcebergRegistrator implements KryoRegistrator { @Override public void registerClasses(Kryo kryo) { try { kryo.register(Class.forName("org.apache.iceberg.GenericDataFile"), new JavaSerializer()); } catch (ClassNotFoundException e) { throw new RuntimeException(e); } } } SparkSession.builder() .config("spark.kryo.registrator", "org.apache.spark.sql.serializer.IcebergRegistrator") ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
