This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 8163c26 [SPARK-31010][SQL][FOLLOW-UP] Give an example for typed Scala
UDF in error message
8163c26 is described below
commit 8163c267b0dc4d117b22936e0371f53e1b4620b6
Author: yi.wu <[email protected]>
AuthorDate: Thu Mar 12 21:16:02 2020 +0900
[SPARK-31010][SQL][FOLLOW-UP] Give an example for typed Scala UDF in error
message
In the error message, adding an example for typed Scala UDF.
Help user to know how to migrate to typed Scala UDF.
No, it's a new error message in Spark 3.0.
Pass Jenkins.
Closes #27884 from Ngone51/spark_31010_followup.
Authored-by: yi.wu <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
---
.../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 2 +-
sql/core/src/main/scala/org/apache/spark/sql/functions.scala | 5 +++--
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 644fe89..3738922 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -2032,7 +2032,7 @@ object SQLConf {
buildConf("spark.sql.legacy.allowUntypedScalaUDF")
.internal()
.doc("When set to true, user is allowed to use
org.apache.spark.sql.functions." +
- "udf(f: AnyRef, dataType: DataType). Otherwise, exception will be
throw.")
+ "udf(f: AnyRef, dataType: DataType). Otherwise, an exception will be
thrown at runtime.")
.booleanConf
.createWithDefault(false)
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
index f280ec3..e2d3d55 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
@@ -4740,8 +4740,9 @@ object functions {
"information. Spark may blindly pass null to the Scala closure with
primitive-type " +
"argument, and the closure will see the default value of the Java type
for the null " +
"argument, e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for
null input. " +
- "You could use other typed Scala UDF APIs to avoid this problem, or
set " +
- s"${SQLConf.LEGACY_ALLOW_UNTYPED_SCALA_UDF.key} to true and use this
API with caution."
+ "You could use typed Scala UDF APIs (e.g. `udf((x: Int) => x)`) to
avoid this problem, " +
+ s"or set ${SQLConf.LEGACY_ALLOW_UNTYPED_SCALA_UDF.key} to true and use
this API with " +
+ s"caution."
throw new AnalysisException(errorMsg)
}
SparkUserDefinedFunction(f, dataType, inputSchemas = Nil)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
