Repository: spark Updated Branches: refs/heads/master 3de24ae2e -> 48ee16d80
[SPARK-14055] writeLocksByTask need to be update when removeBlock ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-14055 ## How was this patch tested? manual tests by running LiveJournalPageRank on a large dataset ( the dataset must larger enough to incure RDD partition eviction). Author: Ernest <[email protected]> Closes #11875 from Earne/issue-14055. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/48ee16d8 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/48ee16d8 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/48ee16d8 Branch: refs/heads/master Commit: 48ee16d8012602c75d50aa2a85e26b7de3c48944 Parents: 3de24ae Author: Ernest <[email protected]> Authored: Wed Mar 23 10:29:36 2016 -0700 Committer: Josh Rosen <[email protected]> Committed: Wed Mar 23 10:29:36 2016 -0700 ---------------------------------------------------------------------- core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala | 1 + 1 file changed, 1 insertion(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/48ee16d8/core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala b/core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala index 94d11c5..ca53534 100644 --- a/core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala +++ b/core/src/main/scala/org/apache/spark/storage/BlockInfoManager.scala @@ -421,6 +421,7 @@ private[storage] class BlockInfoManager extends Logging { infos.remove(blockId) blockInfo.readerCount = 0 blockInfo.writerTask = BlockInfo.NO_WRITER + writeLocksByTask.removeBinding(currentTaskAttemptId, blockId) } case None => throw new IllegalArgumentException( --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
