This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7ce4dc64273 [SPARK-41775][PYTHON][FOLLOWUP] Use pyspark.cloudpickle
instead of `cloudpickle` in torch distributor
7ce4dc64273 is described below
commit 7ce4dc642736d78e79dff8e0b671cfd1b5d44166
Author: Weichen Xu <[email protected]>
AuthorDate: Sat May 27 09:06:39 2023 -0700
[SPARK-41775][PYTHON][FOLLOWUP] Use pyspark.cloudpickle instead of
`cloudpickle` in torch distributor
### What changes were proposed in this pull request?
Use pyspark.cloudpickle instead of `cloudpickle` in torch distributor
### Why are the changes needed?
Make ser and deser code consistent
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Closes #41337 from WeichenXu123/fix-torch-distributor-import-cloudpickle.
Authored-by: Weichen Xu <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/pyspark/ml/torch/distributor.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/ml/torch/distributor.py
b/python/pyspark/ml/torch/distributor.py
index 8a41bdcc886..ad8b4d8cc25 100644
--- a/python/pyspark/ml/torch/distributor.py
+++ b/python/pyspark/ml/torch/distributor.py
@@ -814,7 +814,7 @@ class TorchDistributor(Distributor):
) -> str:
code = textwrap.dedent(
f"""
- import cloudpickle
+ from pyspark import cloudpickle
import os
if __name__ == "__main__":
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
