This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3ab0f2de54af [SPARK-51675][SS][FOLLOW-UP] Clear internal maps on close
to remove DB open dep in other places
3ab0f2de54af is described below
commit 3ab0f2de54af19daf09450ccad4fe79dac075cab
Author: Anish Shrigondekar <anish.shrigonde...@databricks.com>
AuthorDate: Fri Apr 4 10:41:17 2025 +0900
[SPARK-51675][SS][FOLLOW-UP] Clear internal maps on close to remove DB open
dep in other places
### What changes were proposed in this pull request?
Clear internal maps on close to remove DB open dep in other places
### Why are the changes needed?
Refactor related change
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Verified that tests failed without db open check and not clearing the
internal state.
```
===== POSSIBLE THREAD LEAK IN SUITE
o.a.s.sql.execution.streaming.state.RocksDBSuite, threads: rpc-boss-3-1
(daemon=true), ForkJoinPool.commonPool-worker-2 (daemon=true), shuffle-boss-6-1
(daemon=true), ForkJoinPool.commonPool-worker-1 (daemon=true) =====
[info] Run completed in 11 seconds.
[info] Total number of tests run: 16
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 16, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #50510 from anishshri-db/task/SPARK-51715.
Authored-by: Anish Shrigondekar <anish.shrigonde...@databricks.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensou...@gmail.com>
---
.../org/apache/spark/sql/execution/streaming/state/RocksDB.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
index b324dc73f529..0fbb2c0ff367 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
@@ -267,7 +267,7 @@ class RocksDB(
* @return - true if the column family exists, false otherwise
*/
def checkColFamilyExists(colFamilyName: String): Boolean = {
- db != null && colFamilyNameToInfoMap.containsKey(colFamilyName)
+ colFamilyNameToInfoMap.containsKey(colFamilyName)
}
// This method sets the internal column family metadata to
@@ -1215,6 +1215,8 @@ class RocksDB(
}
silentDeleteRecursively(localRootDir, "closing RocksDB")
+ // Clear internal maps to reset the state
+ clearColFamilyMaps()
} catch {
case e: Exception =>
logWarning("Error closing RocksDB", e)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
