Repository: spark Updated Branches: refs/heads/branch-1.1 e4f5695ec -> aa9ebdaa2
[SPARK-4467] Partial fix for fetch failure in sort-based shuffle (1.1) This is the 1.1 version of #3302. There has been some refactoring in master so we can't cherry-pick that PR. Author: Andrew Or <[email protected]> Closes #3330 from andrewor14/sort-fetch-fail and squashes the following commits: 486fc49 [Andrew Or] Reset `elementsRead` Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/aa9ebdaa Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/aa9ebdaa Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/aa9ebdaa Branch: refs/heads/branch-1.1 Commit: aa9ebdaa28bebccc8f65a323d7c6fc34cf68ef73 Parents: e4f5695 Author: Andrew Or <[email protected]> Authored: Mon Nov 17 18:10:49 2014 -0800 Committer: Andrew Or <[email protected]> Committed: Mon Nov 17 18:10:49 2014 -0800 ---------------------------------------------------------------------- .../scala/org/apache/spark/util/collection/ExternalSorter.scala | 1 + 1 file changed, 1 insertion(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/aa9ebdaa/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala index 3136306..97ddd96 100644 --- a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala +++ b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala @@ -288,6 +288,7 @@ private[spark] class ExternalSorter[K, V, C]( myMemoryThreshold = 0 _memoryBytesSpilled += memorySize + elementsRead = 0 } /** --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
