This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3c5f6a2ebf8 [SPARK-44625][CONNECT][FOLLOWUP] Make initialization of
SparkConnectExecutionManager lazy
3c5f6a2ebf8 is described below
commit 3c5f6a2ebf8aaa1fa9081793ced9a1a76c66c371
Author: Juliusz Sompolski <[email protected]>
AuthorDate: Sat Aug 12 08:48:02 2023 +0900
[SPARK-44625][CONNECT][FOLLOWUP] Make initialization of
SparkConnectExecutionManager lazy
### What changes were proposed in this pull request?
SparkConnectExecutionManager is a val field in SparkConnectService, so it
gets initialized the first time SparkConnectService object is touched. In some
scenarios, this can accidentally happen too early, before anything is
initialized. SparkConnectExecutionManager wants to read configs from SparkEnv,
and when it's not initialized yet it can fail.
Make SparkConnectExecutionManager lazy. Since it is `private[connect]` it
should not then be initialized before the service is started.
Raised https://issues.apache.org/jira/browse/SPARK-44779 as a followup to
look at the Spark Connect initialization in general.
### Why are the changes needed?
Fix initialization order.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Got one test failure in one specific environment that failed because of
wrong initialization order.
Closes #42457 from juliuszsompolski/SPARK-44625-fup.
Authored-by: Juliusz Sompolski <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../org/apache/spark/sql/connect/service/SparkConnectService.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
index a7b5fbdcec0..fe773e4b704 100644
---
a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
+++
b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
@@ -278,7 +278,7 @@ object SparkConnectService extends Logging {
private val userSessionMapping =
cacheBuilder(CACHE_SIZE, CACHE_TIMEOUT_SECONDS).build[SessionCacheKey,
SessionHolder]()
- private[connect] val executionManager = new SparkConnectExecutionManager()
+ private[connect] lazy val executionManager = new
SparkConnectExecutionManager()
private[connect] val streamingSessionManager =
new SparkConnectStreamingQueryCache()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
