This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new d60a40e5d30 [SPARK-45167][CONNECT][PYTHON][FOLLOW-UP][3.5] Use lighter
threading Rlock
d60a40e5d30 is described below
commit d60a40e5d30e565536e0833fc650dc581439ee89
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Oct 5 13:00:02 2023 +0900
[SPARK-45167][CONNECT][PYTHON][FOLLOW-UP][3.5] Use lighter threading Rlock
### What changes were proposed in this pull request?
This PR reverts the revert:
https://github.com/apache/spark/commit/522af69713502d33d34b330bce6166e3d15dba8a.
It only partially ports the real change within main code. It excludes the
testing side which depends on
https://github.com/apache/spark/commit/9798244ca647ec68d36f4b9b21356a6de5f73f77
that does not exist in `branch-3.5`.
### Why are the changes needed?
Mainly for code clean-up.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests should cover this.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43221 from HyukjinKwon/SPARK-45167-followup2.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/connect/client/reattach.py | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/connect/client/reattach.py
b/python/pyspark/sql/connect/client/reattach.py
index e58864b965b..6addb5bd2c6 100644
--- a/python/pyspark/sql/connect/client/reattach.py
+++ b/python/pyspark/sql/connect/client/reattach.py
@@ -18,12 +18,11 @@ from pyspark.sql.connect.utils import check_dependencies
check_dependencies(__name__)
+from threading import RLock
import warnings
import uuid
from collections.abc import Generator
from typing import Optional, Dict, Any, Iterator, Iterable, Tuple, Callable,
cast, Type, ClassVar
-from multiprocessing import RLock
-from multiprocessing.synchronize import RLock as RLockBase
from multiprocessing.pool import ThreadPool
import os
@@ -56,7 +55,7 @@ class ExecutePlanResponseReattachableIterator(Generator):
"""
# Lock to manage the pool
- _lock: ClassVar[RLockBase] = RLock()
+ _lock: ClassVar[RLock] = RLock()
_release_thread_pool: Optional[ThreadPool] = ThreadPool(os.cpu_count() if
os.cpu_count() else 8)
@classmethod
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
