Repository: spark Updated Branches: refs/heads/branch-2.0 a99ea4c9e -> 744aac8e6
[MINOR][DOC] Add an up-to-date description for default serialization during shuffling ## What changes were proposed in this pull request? This PR aims to make the doc up-to-date. The documentation is generally correct, but after https://issues.apache.org/jira/browse/SPARK-13926, Spark starts to choose Kyro as a default serialization library during shuffling of simple types, arrays of simple types, or string type. ## How was this patch tested? This is a documentation update. Author: Dongjoon Hyun <[email protected]> Closes #15315 from dongjoon-hyun/SPARK-DOC-SERIALIZER. (cherry picked from commit 15e9bbb49e00b3982c428d39776725d0dea2cdfa) Signed-off-by: Reynold Xin <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/744aac8e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/744aac8e Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/744aac8e Branch: refs/heads/branch-2.0 Commit: 744aac8e6ff04d7a3f1e8ccad335605ac8fe2f29 Parents: a99ea4c Author: Dongjoon Hyun <[email protected]> Authored: Fri Sep 30 22:05:59 2016 -0700 Committer: Reynold Xin <[email protected]> Committed: Fri Sep 30 22:06:07 2016 -0700 ---------------------------------------------------------------------- docs/tuning.md | 1 + 1 file changed, 1 insertion(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/744aac8e/docs/tuning.md ---------------------------------------------------------------------- diff --git a/docs/tuning.md b/docs/tuning.md index cbf3721..9c43b31 100644 --- a/docs/tuning.md +++ b/docs/tuning.md @@ -45,6 +45,7 @@ and calling `conf.set("spark.serializer", "org.apache.spark.serializer.KryoSeria This setting configures the serializer used for not only shuffling data between worker nodes but also when serializing RDDs to disk. The only reason Kryo is not the default is because of the custom registration requirement, but we recommend trying it in any network-intensive application. +Since Spark 2.0.0, we internally use Kryo serializer when shuffling RDDs with simple types, arrays of simple types, or string type. Spark automatically includes Kryo serializers for the many commonly-used core Scala classes covered in the AllScalaRegistrar from the [Twitter chill](https://github.com/twitter/chill) library. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
