This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new beedbda4a27f [SPARK-54146][CORE][SQL] Clean up the usage of deprecated
Jackson API
beedbda4a27f is described below
commit beedbda4a27f72bad07c64a0f69c4db3fa2cf5c9
Author: yangjie01 <[email protected]>
AuthorDate: Mon Nov 3 22:08:23 2025 +0800
[SPARK-54146][CORE][SQL] Clean up the usage of deprecated Jackson API
### What changes were proposed in this pull request?
This PR cleans up the usage of deprecated Jackson APIs as follows:
```
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation,
site=org.apache.spark.ErrorClassesJsonReader.readAsMap.map,
origin=com.fasterxml.jackson.databind.ObjectMapper.readValue
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation,
site=org.apache.spark.status.KVUtils.KVStoreScalaSerializer,
origin=com.fasterxml.jackson.databind.ObjectMapper.setSerializationInclusion
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation,
site=org.apache.spark.status.api.v1.JacksonMessageWriter,
origin=com.fasterxml.jackson.databind.ObjectMapper.setSerializationInclusion
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation, site=org.apache.spark.SparkThrowableSuite,
origin=com.fasterxml.jackson.databind.ObjectMapper.readValue
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation, site=org.apache.spark.SparkThrowableSuite,
origin=com.fasterxml.jackson.databind.ObjectMapper.setSerializationInclusion
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation, site=org.apache.spark.SparkThrowableSuite,
origin=com.fasterxml.jackson.databind.ObjectMapper.readValue
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation, site=org.apache.spark.SparkThrowableSuite,
origin=com.fasterxml.jackson.databind.ObjectMapper.readValue
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation,
site=org.apache.spark.sql.catalyst.util.RebaseDateTime.loadRebaseRecords.jsonRebaseRecords,
origin=com.fasterxml.jackson.module.scala.ClassTagExtensions.readValue,
version=2.20.0
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation,
site=org.apache.spark.sql.catalyst.catalog.UserDefinedFunction.getObjectMapper,
origin=com.fasterxml.jackson.databind.ObjectMapper.setSerializationInclusion
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation,
site=org.apache.spark.sql.catalyst.catalog.ClusterBySpec.mapper,
origin=com.fasterxml.jackson.databind.ObjectMapper.setSerializationInclusion
Applicable -Wconf / nowarn filters for this warning: msg=<part of the
message>, cat=deprecation,
site=org.apache.spark.sql.execution.streaming.state.RocksDBCheckpointMetadata.mapper,
origin=com.fasterxml.jackson.databind.ObjectMapper.setSerializationInclusion
```
Relevant modifications reference:
1. Replace `T readValue(URL src, TypeReference<T> valueTypeRef)` with `T
readValue(InputStream src, TypeReference<T> valueTypeRef)`
https://github.com/FasterXML/jackson-databind/blob/4260f88180e5e45f3be1a290114e55c042bb2213/src/main/java/com/fasterxml/jackson/databind/ObjectMapper.java#L3848-L3877
```
/**
* Method to deserialize JSON content from given resource into given
Java type.
*<p>
* NOTE: handling of {link java.net.URL} is delegated to
* {link JsonFactory#createParser(java.net.URL)} and usually simply
* calls {link java.net.URL#openStream()}, meaning no special handling
* is done. If different HTTP connection options are needed you will
need
* to create {link java.io.InputStream} separately.
*
* throws IOException if a low-level I/O problem (unexpected
end-of-input,
* network error) occurs (passed through as-is without additional
wrapping -- note
* that this is one case where {link
DeserializationFeature#WRAP_EXCEPTIONS}
* does NOT result in wrapping of exception even if enabled)
* throws StreamReadException if underlying input contains invalid
content
* of type {link JsonParser} supports (JSON for default case)
* throws DatabindException if the input JSON structure does not match
structure
* expected for result type (or has other mismatch issues)
*
* deprecated since 2.20 deprecated as it calls {link
JsonFactory#createParser(URL)}.
* Instead, use equivalent methods that take InputStream
inputs instead.
*/
Deprecated // since 2.20
SuppressWarnings("unchecked")
public <T> T readValue(URL src, Class<T> valueType)
throws IOException, StreamReadException, DatabindException
{
_assertNotNull("src", src);
return (T) _readMapAndClose(_jsonFactory.createParser(src),
_typeFactory.constructType(valueType));
}
```
2. Replace `ObjectMapper setSerializationInclusion(JsonInclude.Include
incl)` with `ObjectMapper setDefaultPropertyInclusion(JsonInclude.Value incl)`
https://github.com/FasterXML/jackson-databind/blob/4260f88180e5e45f3be1a290114e55c042bb2213/src/main/java/com/fasterxml/jackson/databind/ObjectMapper.java#L1853-L1872
```
/**
* Convenience method, equivalent to calling:
*<pre>
* setPropertyInclusion(JsonInclude.Value.construct(incl, incl));
*</pre>
*<p>
* NOTE: behavior differs slightly from 2.8, where second argument was
* implied to be <code>JsonInclude.Include.ALWAYS</code>.
*<p>
* NOTE: in Jackson 3.x all configuration goes through {code
ObjectMapper} builders,
* see {link com.fasterxml.jackson.databind.cfg.MapperBuilder},
* and this method will be removed from 3.0.
*
* deprecated Since 2.9 use {link
#setDefaultPropertyInclusion(JsonInclude.Include)}
*/
Deprecated
public ObjectMapper setSerializationInclusion(JsonInclude.Include incl)
{
setPropertyInclusion(JsonInclude.Value.construct(incl, incl));
return this;
}
```
### Why are the changes needed?
To clean up the usage of deprecated APIs and align with Jackson's
recommended practices.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass Github Actions
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #52845 from LuciferYang/jackson-deprecation.
Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
(cherry picked from commit 46e7bfb58141bea9c51066d789bb228373977e9c)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala | 2 +-
core/src/main/scala/org/apache/spark/status/KVUtils.scala | 2 +-
.../org/apache/spark/status/api/v1/JacksonMessageWriter.scala | 2 +-
core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala | 9 +++++----
.../org/apache/spark/sql/catalyst/util/RebaseDateTime.scala | 2 +-
.../apache/spark/sql/catalyst/catalog/UserDefinedFunction.scala | 2 +-
.../scala/org/apache/spark/sql/catalyst/catalog/interface.scala | 2 +-
.../spark/sql/execution/streaming/state/RocksDBFileManager.scala | 2 +-
8 files changed, 12 insertions(+), 11 deletions(-)
diff --git
a/common/utils/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala
b/common/utils/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala
index af76056dfe92..0d958e3f7160 100644
--- a/common/utils/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala
+++ b/common/utils/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala
@@ -152,7 +152,7 @@ private object ErrorClassesJsonReader {
.addModule(DefaultScalaModule)
.build()
private def readAsMap(url: URL): Map[String, ErrorInfo] = {
- val map = mapper.readValue(url, new TypeReference[Map[String,
ErrorInfo]]() {})
+ val map = mapper.readValue(url.openStream(), new TypeReference[Map[String,
ErrorInfo]]() {})
val errorClassWithDots = map.collectFirst {
case (errorClass, _) if errorClass.contains('.') => errorClass
case (_, ErrorInfo(_, Some(map), _, _)) if
map.keys.exists(_.contains('.')) =>
diff --git a/core/src/main/scala/org/apache/spark/status/KVUtils.scala
b/core/src/main/scala/org/apache/spark/status/KVUtils.scala
index 76fb654f8da2..1a9d36f6c647 100644
--- a/core/src/main/scala/org/apache/spark/status/KVUtils.scala
+++ b/core/src/main/scala/org/apache/spark/status/KVUtils.scala
@@ -74,7 +74,7 @@ private[spark] object KVUtils extends Logging {
private[spark] class KVStoreScalaSerializer extends KVStoreSerializer {
mapper.registerModule(DefaultScalaModule)
- mapper.setSerializationInclusion(JsonInclude.Include.NON_ABSENT)
+ mapper.setDefaultPropertyInclusion(JsonInclude.Include.NON_ABSENT)
}
diff --git
a/core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala
b/core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala
index 259d0aacc575..1103f5529782 100644
---
a/core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala
+++
b/core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala
@@ -48,7 +48,7 @@ private[v1] class JacksonMessageWriter extends
MessageBodyWriter[Object]{
}
mapper.registerModule(com.fasterxml.jackson.module.scala.DefaultScalaModule)
mapper.enable(SerializationFeature.INDENT_OUTPUT)
- mapper.setSerializationInclusion(JsonInclude.Include.NON_ABSENT)
+ mapper.setDefaultPropertyInclusion(JsonInclude.Include.NON_ABSENT)
mapper.setDateFormat(JacksonMessageWriter.makeISODateFormat)
override def isWriteable(
diff --git a/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
b/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
index d182bd165f1f..24a1fa740175 100644
--- a/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
@@ -75,7 +75,8 @@ class SparkThrowableSuite extends SparkFunSuite {
.addModule(DefaultScalaModule)
.enable(STRICT_DUPLICATE_DETECTION)
.build()
- mapper.readValue(errorJsonFilePath.toUri.toURL, new
TypeReference[Map[String, ErrorInfo]]() {})
+ mapper.readValue(
+ errorJsonFilePath.toUri.toURL.openStream(), new
TypeReference[Map[String, ErrorInfo]]() {})
}
test("Error conditions are correctly formatted") {
@@ -88,7 +89,7 @@ class SparkThrowableSuite extends SparkFunSuite {
val prettyPrinter = new DefaultPrettyPrinter()
.withArrayIndenter(DefaultIndenter.SYSTEM_LINEFEED_INSTANCE)
val rewrittenString =
mapper.configure(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS, true)
- .setSerializationInclusion(Include.NON_ABSENT)
+ .setDefaultPropertyInclusion(Include.NON_ABSENT)
.writer(prettyPrinter)
.writeValueAsString(errorReader.errorInfoMap)
@@ -124,9 +125,9 @@ class SparkThrowableSuite extends SparkFunSuite {
.enable(STRICT_DUPLICATE_DETECTION)
.build()
val errorClasses = mapper.readValue(
- errorClassesJson, new TypeReference[Map[String, String]]() {})
+ errorClassesJson.openStream(), new TypeReference[Map[String, String]]()
{})
val errorStates = mapper.readValue(
- errorStatesJson, new TypeReference[Map[String, ErrorStateInfo]]() {})
+ errorStatesJson.openStream(), new TypeReference[Map[String,
ErrorStateInfo]]() {})
val errorConditionStates =
errorReader.errorInfoMap.values.toSeq.flatMap(_.sqlState).toSet
assert(Set("22012", "22003", "42601").subsetOf(errorStates.keySet))
assert(errorClasses.keySet.filter(!_.matches("[A-Z0-9]{2}")).isEmpty)
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala
index 8dff1ceccfcf..ca8e73a517d5 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala
@@ -288,7 +288,7 @@ object RebaseDateTime {
// `JsonRebaseRecord`. Mutable HashMap is used here instead of AnyRefMap due
to SPARK-49491.
private[sql] def loadRebaseRecords(fileName: String): HashMap[String,
RebaseInfo] = {
val file = SparkClassUtils.getSparkClassLoader.getResource(fileName)
- val jsonRebaseRecords = mapper.readValue[Seq[JsonRebaseRecord]](file)
+ val jsonRebaseRecords =
mapper.readValue[Seq[JsonRebaseRecord]](file.openStream())
val hashMap = new HashMap[String, RebaseInfo]
hashMap.sizeHint(jsonRebaseRecords.size)
jsonRebaseRecords.foreach { jsonRecord =>
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/UserDefinedFunction.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/UserDefinedFunction.scala
index 8ed241468352..3365b11b0742 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/UserDefinedFunction.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/UserDefinedFunction.scala
@@ -132,7 +132,7 @@ object UserDefinedFunction {
*/
private def getObjectMapper: ObjectMapper = {
val mapper = new ObjectMapper with ClassTagExtensions
- mapper.setSerializationInclusion(Include.NON_ABSENT)
+ mapper.setDefaultPropertyInclusion(Include.NON_ABSENT)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
mapper.registerModule(DefaultScalaModule)
mapper
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
index 64d816587f4d..07d26813be94 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
@@ -278,7 +278,7 @@ case class ClusterBySpec(columnNames: Seq[NamedReference]) {
object ClusterBySpec {
private val mapper = {
val ret = new ObjectMapper() with ClassTagExtensions
- ret.setSerializationInclusion(Include.NON_ABSENT)
+ ret.setDefaultPropertyInclusion(Include.NON_ABSENT)
ret.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
ret.registerModule(DefaultScalaModule)
ret
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
index 7bef692e264a..2e86ff70d58f 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
@@ -1123,7 +1123,7 @@ object RocksDBCheckpointMetadata {
/** Used to convert between classes and JSON. */
lazy val mapper = {
val _mapper = new ObjectMapper with ClassTagExtensions
- _mapper.setSerializationInclusion(Include.NON_ABSENT)
+ _mapper.setDefaultPropertyInclusion(Include.NON_ABSENT)
_mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
_mapper.registerModule(DefaultScalaModule)
_mapper
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
