sunchao commented on code in PR #5606:
URL: https://github.com/apache/datafusion-comet/pull/5606#discussion_r3901095367
##########
spark/src/main/scala/org/apache/comet/serde/strings.scala:
##########
@@ -109,15 +109,37 @@ object CometOctetLength extends
CometScalarFunction[OctetLength]("octet_length")
}
}
-object CometStringTranslate extends
CometScalarFunction[StringTranslate]("translate") {
+object CometStringTranslate
+ extends CometScalarFunction[StringTranslate]("translate")
+ with NativeOptInAvailable {
private val incompatReason =
"DataFusion's translate iterates over Unicode graphemes (Spark uses code
points) and" +
" substitutes U+0000 instead of treating it as a deletion sentinel"
override def getIncompatibleReasons(): Seq[String] = Seq(incompatReason)
- override def getSupportLevel(expr: StringTranslate): SupportLevel =
Incompatible(
- Some(incompatReason))
+ override def getSupportLevel(expr: StringTranslate): SupportLevel =
+ if (!CometConf.isExprAllowIncompat(getExprConfigName(expr))) {
+ Compatible(nativeOptIn =
Review Comment:
[P2] Update the default translate SQL fixture
Could you update the default cases in `string_translate.sql` along with this
change? It still has four `expect_fallback(is not fully compatible with Spark)`
blocks. The SQL-file suite registers these blocks and requires that fallback
contract, but this `Compatible` branch now accepts the first ordinary
three-string-column query and dispatches it by default. The successful new
route therefore cannot satisfy the old fixture expectation, even when its
result values are correct. Please check Comet execution and result parity for
the default cases, and keep any dispatcher-disabled fallback case separate with
its actual reason. This is source-derived, not an observed test failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
