mbutrovich opened a new issue, #4786:
URL: https://github.com/apache/datafusion-comet/issues/4786
### Describe the bug
For a null operand against an empty IN list, Comet always returns `false`.
Spark returns `null` when `spark.sql.legacy.nullInEmptyListBehavior=true`.
Comet ignores the config and the null operand.
### Steps to reproduce
Empty IN lists are not expressible in SQL, so build via the DataFrame API;
`ConvertToLocalRelation` and `OptimizeIn` must be disabled or the IN is folded
away. Add to a suite extending `CometTestBase` (config exists in Spark 4+):
```scala
test("null IN empty list under legacy behavior") {
assume(org.apache.comet.CometSparkSessionExtensions.isSpark40Plus)
withSQLConf(
"spark.sql.optimizer.excludedRules" ->
("org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation," +
"org.apache.spark.sql.catalyst.optimizer.OptimizeIn"),
"spark.sql.legacy.nullInEmptyListBehavior" -> "true") {
val data: Seq[(Integer, Integer)] =
Seq((Integer.valueOf(1), Integer.valueOf(1)), (null,
Integer.valueOf(2)))
withParquetTable(data, "t") {
val df = sql("SELECT _1 AS a FROM t").select(col("a"), col("a").isin())
checkSparkAnswer(df)
}
}
}
```
```
== Results ==
!== Spark Answer - 2 == == Comet Answer - 2 ==
struct<a:int,(a IN ()):boolean> struct<a:int,(a IN ()):boolean>
[1,false] [1,false]
![null,null] [null,false]
```
### Expected behavior
`null IN ()` returns `null` under legacy behavior, matching Spark. (Under
ANSI-on, the Spark 4 default, `false` is expected and Comet already matches, so
only the legacy path is affected.)
### Additional context
Found while enabling `CometLocalTableScanExec` by default (#4393), but
reproduces over a Parquet scan. Upstream test: `EmptyInSuite` "IN with empty
list".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
