Repository: spark
Updated Branches:
refs/heads/master 14a32a647 -> 9df08e218
[SPARK-22454][CORE] ExternalShuffleClient.close() should check clientFactory
null
## What changes were proposed in this pull request?
`ExternalShuffleClient.close()` should check `clientFactory` null. otherwise it
will throw NPE sometimes:
```
17/11/06 20:08:05 ERROR Utils: Uncaught exception in thread main
java.lang.NullPointerException
at
org.apache.spark.network.shuffle.ExternalShuffleClient.close(ExternalShuffleClient.java:152)
at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1407)
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:89)
at
org.apache.spark.SparkContext$$anonfun$stop$11.apply$mcV$sp(SparkContext.scala:1849)
```
## How was this patch tested?
manual tests
Author: Yuming Wang <[email protected]>
Closes #19670 from wangyum/SPARK-22454.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9df08e21
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9df08e21
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9df08e21
Branch: refs/heads/master
Commit: 9df08e218cfd4dd91bc407b98528b74f452f34f8
Parents: 14a32a6
Author: Yuming Wang <[email protected]>
Authored: Tue Nov 7 08:30:58 2017 +0000
Committer: Sean Owen <[email protected]>
Committed: Tue Nov 7 08:30:58 2017 +0000
----------------------------------------------------------------------
.../org/apache/spark/network/shuffle/ExternalShuffleClient.java | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/9df08e21/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleClient.java
----------------------------------------------------------------------
diff --git
a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleClient.java
b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleClient.java
index 510017f..7ed0b6e 100644
---
a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleClient.java
+++
b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleClient.java
@@ -148,6 +148,9 @@ public class ExternalShuffleClient extends ShuffleClient {
@Override
public void close() {
checkInit();
- clientFactory.close();
+ if (clientFactory != null) {
+ clientFactory.close();
+ clientFactory = null;
+ }
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
