Enrico Minack created SPARK-52506:
-------------------------------------
Summary: Migrate to fallback storage only
Key: SPARK-52506
URL: https://issues.apache.org/jira/browse/SPARK-52506
Project: Spark
Issue Type: Sub-task
Components: k8s, Kubernetes
Affects Versions: 4.1.0
Reporter: Enrico Minack
Using the storage decommissioning feature with fallback storage on Kubernetes
allows to migrate shuffle data from executors onto an external cloud storage.
Currently, this fallback storage can only be used in conjunction with migrating
shuffle data to other executors. However, those executors are also subject of
eventual decommissioning, so shuffle data are potentially migrated multiple
times.
The user should have the option to migrate shuffle data at most once, and that
is to the fallback storage only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
