This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ce57a3d13154 [SPARK-50067][SPARK-50066][SPARK-49954][SQL][FOLLOWUP]
Make returnNullable as false
ce57a3d13154 is described below
commit ce57a3d131545bd18661cc839d894f59a47047b8
Author: panbingkun <[email protected]>
AuthorDate: Thu Nov 28 11:13:05 2024 +0800
[SPARK-50067][SPARK-50066][SPARK-49954][SQL][FOLLOWUP] Make returnNullable
as false
### What changes were proposed in this pull request?
The pr is following up
[schema_of_json](https://github.com/apache/spark/pull/48473),
[schema_of_xml](https://github.com/apache/spark/pull/48594) and
[schema_of_csv](https://github.com/apache/spark/pull/48595), to make
returnNullable as false.
### Why are the changes needed?
As `cloud-fan`'s comment
https://github.com/apache/spark/pull/48594/files#r1860534460, we should follow
the original logic, otherwise it's a regression.
https://github.com/apache/spark/blob/1a502d32ef5a69739e10b827be4c9063b2a20493/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L846
https://github.com/apache/spark/blob/1a502d32ef5a69739e10b827be4c9063b2a20493/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala#L166
https://github.com/apache/spark/blob/1a502d32ef5a69739e10b827be4c9063b2a20493/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala#L141
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48987 from panbingkun/SPARK-50066_FOLLOWUP.
Authored-by: panbingkun <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
.../org/apache/spark/sql/catalyst/expressions/csvExpressions.scala | 3 ++-
.../org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala | 4 ++--
.../org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala | 4 ++--
3 files changed, 6 insertions(+), 5 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
index 02e5488835c9..bf3d8eee9ed5 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
@@ -173,7 +173,8 @@ case class SchemaOfCsv(
"evaluate",
dataType,
Seq(child),
- Seq(child.dataType))
+ Seq(child.dataType),
+ returnNullable = false)
}
/**
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
index 0381d1e712ae..68cce1c2a138 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
@@ -886,8 +886,8 @@ case class SchemaOfJson(
Literal(jsonOptions, jsonOptionsObjectType),
Literal(jsonInferSchema, jsonInferSchemaObjectType),
child),
- Seq(jsonFactoryObjectType, jsonOptionsObjectType,
jsonInferSchemaObjectType, child.dataType)
- )
+ Seq(jsonFactoryObjectType, jsonOptionsObjectType,
jsonInferSchemaObjectType, child.dataType),
+ returnNullable = false)
override def prettyName: String = "schema_of_json"
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
index 6f004cbce426..66f7f25e4abe 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
@@ -208,8 +208,8 @@ case class SchemaOfXml(
dataType,
"schemaOfXml",
Seq(Literal(xmlInferSchema, xmlInferSchemaObjectType), child),
- Seq(xmlInferSchemaObjectType, child.dataType)
- )
+ Seq(xmlInferSchemaObjectType, child.dataType),
+ returnNullable = false)
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
