kazuyukitanimura commented on code in PR #509:
URL: https://github.com/apache/datafusion-comet/pull/509#discussion_r1630801139
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -701,17 +695,17 @@ object QueryPlanSerde extends Logging with
ShimQueryPlanSerde with CometExprShim
case UnaryExpression(child) if expr.prettyName == "trycast" =>
val timeZoneId = SQLConf.get.sessionLocalTimeZone
- handleCast(child, inputs, expr.dataType, Some(timeZoneId), "TRY")
+ handleCast(child, inputs, expr.dataType, Some(timeZoneId),
CometEvalMode.TRY)
- case Cast(child, dt, timeZoneId, evalMode) =>
- val evalModeStr = if (evalMode.isInstanceOf[Boolean]) {
- // Spark 3.2 & 3.3 has ansiEnabled boolean
- if (evalMode.asInstanceOf[Boolean]) "ANSI" else "LEGACY"
- } else {
- // Spark 3.4+ has EvalMode enum with values LEGACY, ANSI, and TRY
- evalMode.toString
- }
- handleCast(child, inputs, dt, timeZoneId, evalModeStr)
+ case c @ Cast(child, dt, timeZoneId, _) =>
+ handleCast(child, inputs, dt, timeZoneId, evalMode(c))
+
+ case expr if isUnsupportedAnsiExpr(expr) &&
!CometConf.COMET_ANSI_MODE_ENABLED.get() =>
Review Comment:
I feel `!CometConf.COMET_ANSI_MODE_ENABLED.get()` condition needs to be
removed.
When `COMET_ANSI_MODE_ENABLED=true`, we still need to fallback to Spark if
the expression is not ANSI compatible
##########
spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala:
##########
@@ -69,7 +69,7 @@ abstract class CometTestBase
conf.set("spark.hadoop.fs.file.impl", classOf[DebugFilesystem].getName)
conf.set("spark.ui.enabled", "false")
conf.set(SQLConf.SHUFFLE_PARTITIONS, 10) // reduce parallelism in tests
- conf.set(SQLConf.ANSI_ENABLED.key, "false")
+ conf.set(CometConf.COMET_ANSI_MODE_ENABLED.key, "true")
Review Comment:
What about leaving `COMET_ANSI_MODE_ENABLED` as default value as well? I.e.
false for 3.x, true for 4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
