This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 9caf009 [SPARK-31055][DOCS] Update config docs for shuffle local host
reads to have dep on external shuffle service
9caf009 is described below
commit 9caf009ecd9041e71efe2ef56ca0e75cc94cb56e
Author: Thomas Graves <[email protected]>
AuthorDate: Mon Mar 9 12:17:59 2020 -0700
[SPARK-31055][DOCS] Update config docs for shuffle local host reads to have
dep on external shuffle service
### What changes were proposed in this pull request?
with SPARK-27651 we now support host local reads for shuffle, but only when
external shuffle service is enabled. Update the config docs to state that.
### Why are the changes needed?
clarify dependency
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
n/a
Closes #27812 from tgravescs/SPARK-27651-follow.
Authored-by: Thomas Graves <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit e807118eef9e0214170ff62c828524d237bd58e3)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/internal/config/package.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 23c31a5..1308a46 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -1135,7 +1135,8 @@ package object config {
private[spark] val SHUFFLE_HOST_LOCAL_DISK_READING_ENABLED =
ConfigBuilder("spark.shuffle.readHostLocalDisk")
- .doc(s"If enabled (and `${SHUFFLE_USE_OLD_FETCH_PROTOCOL.key}` is
disabled), shuffle " +
+ .doc(s"If enabled (and `${SHUFFLE_USE_OLD_FETCH_PROTOCOL.key}` is
disabled and external " +
+ s"shuffle `${SHUFFLE_SERVICE_ENABLED.key}` is enabled), shuffle " +
"blocks requested from those block managers which are running on the
same host are read " +
"from the disk directly instead of being fetched as remote blocks over
the network.")
.booleanConf
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
