This is an automated email from the ASF dual-hosted git repository.
vanzin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from e23c135 [SPARK-29293][BUILD] Move scalafmt to Scala 2.12 profile;
bump to 0.12
add fd2bf55 [SPARK-27651][CORE] Avoid the network when shuffle blocks are
fetched from the same host
No new revisions were added by this update.
Summary of changes:
.../network/shuffle/ExternalBlockHandler.java | 6 +
.../network/shuffle/ExternalBlockStoreClient.java | 49 +++-
.../shuffle/ExternalShuffleBlockResolver.java | 15 ++
.../shuffle/protocol/BlockTransferMessage.java | 4 +-
.../network/shuffle/protocol/BlocksRemoved.java | 2 +-
.../shuffle/protocol/ExecutorShuffleInfo.java | 2 +-
...veBlocks.java => GetLocalDirsForExecutors.java} | 46 ++--
.../shuffle/protocol/LocalDirsForExecutors.java | 117 ++++++++++
.../shuffle/BlockTransferMessagesSuite.java | 24 +-
.../main/scala/org/apache/spark/SparkContext.scala | 8 +
.../org/apache/spark/internal/config/package.scala | 18 ++
.../apache/spark/network/BlockDataManager.scala | 12 +-
.../spark/network/netty/NettyBlockRpcServer.scala | 6 +-
.../spark/shuffle/IndexShuffleBlockResolver.scala | 41 +++-
.../spark/shuffle/ShuffleBlockResolver.scala | 10 +-
.../org/apache/spark/storage/BlockManager.scala | 70 +++++-
.../spark/storage/BlockManagerMasterEndpoint.scala | 26 ++-
.../storage/ShuffleBlockFetcherIterator.scala | 248 +++++++++++++++------
.../apache/spark/ExternalShuffleServiceSuite.scala | 49 +++-
.../netty/NettyBlockTransferSecuritySuite.scala | 2 +-
.../shuffle/BlockStoreShuffleReaderSuite.scala | 3 +-
.../spark/storage/BlockManagerInfoSuite.scala | 1 -
.../apache/spark/storage/BlockManagerSuite.scala | 4 +-
.../storage/ShuffleBlockFetcherIteratorSuite.scala | 220 +++++++++++++-----
24 files changed, 794 insertions(+), 189 deletions(-)
copy
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/{RemoveBlocks.java
=> GetLocalDirsForExecutors.java} (57%)
create mode 100644
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/LocalDirsForExecutors.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
