This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 8e384977f042 [SPARK-49872][FOLLOWUP] Remove Jackson JSON string length
limit in KVStoreScalaSerializer
8e384977f042 is described below
commit 8e384977f04288130c116b7df74141ab13086d88
Author: Kris Mok <[email protected]>
AuthorDate: Wed Jan 7 16:01:19 2026 +0800
[SPARK-49872][FOLLOWUP] Remove Jackson JSON string length limit in
KVStoreScalaSerializer
### What changes were proposed in this pull request?
Along the same lines as https://github.com/apache/spark/pull/52049, apply
the same fix to `KVStoreScalaSerializer` as well.
### Why are the changes needed?
The previous fix in https://github.com/apache/spark/pull/52049 only removed
the Jackson JSON string length limit in `JsonProtocol`, which covered Spark
event <=> JSON SerDe.
However, the exact same problem exists in `KVStoreScalaSerializer` as well,
which is triggered when SHS is configured to use LevelDB/RocksDB KVStores.
Example call chain that triggers the issue:
```
ExecutionPage.render()
→ sqlStore.planGraph(executionId, version)
→ store.read(classOf[SparkPlanGraphWrapper], ...)
→ KVStoreSerializer.deserialize()
→ mapper.readValue() // ← FAILS HERE with 20MB limit
```
### Does this PR introduce _any_ user-facing change?
Yes, users won't hit the Jackson JSON string length limit any more when
using SHS.
### How was this patch tested?
Manually tested with a SQL workload that had an intentionally long string
for the plan graph.
The fix mechanism itself is tested by previous PRs
https://github.com/apache/spark/pull/49163 and
https://github.com/apache/spark/pull/52049, this is just applying the same fix
to SHS as well.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #53711 from rednaxelafx/fix-jackson-strlenlimit-kvstore.
Authored-by: Kris Mok <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit ed201397c2ac3d9143bfb08508bbe7b5fba5f5cd)
Signed-off-by: Wenchen Fan <[email protected]>
---
core/src/main/scala/org/apache/spark/status/KVUtils.scala | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/core/src/main/scala/org/apache/spark/status/KVUtils.scala
b/core/src/main/scala/org/apache/spark/status/KVUtils.scala
index 1a9d36f6c647..f9ea732f9514 100644
--- a/core/src/main/scala/org/apache/spark/status/KVUtils.scala
+++ b/core/src/main/scala/org/apache/spark/status/KVUtils.scala
@@ -25,6 +25,7 @@ import scala.jdk.CollectionConverters._
import scala.reflect.{classTag, ClassTag}
import com.fasterxml.jackson.annotation.JsonInclude
+import com.fasterxml.jackson.core.StreamReadConstraints
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import org.fusesource.leveldbjni.internal.NativeDB
import org.rocksdb.RocksDBException
@@ -76,6 +77,10 @@ private[spark] object KVUtils extends Logging {
mapper.registerModule(DefaultScalaModule)
mapper.setDefaultPropertyInclusion(JsonInclude.Include.NON_ABSENT)
+ // SPARK-49872: Remove jackson JSON string length limitation.
+ mapper.getFactory.setStreamReadConstraints(
+ StreamReadConstraints.builder().maxStringLength(Int.MaxValue).build()
+ )
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
