sunchao commented on PR #4744: URL: https://github.com/apache/datafusion-comet/pull/4744#issuecomment-5744117845
Reviewed `ba319edf9`. **I would not approve yet: the existing short-circuit finding is only partially fixed.** **[P2] Native lambdas still evaluate elements Spark skips** — [CometHighOrderFunction.scala:195](https://github.com/apache/datafusion-comet/blob/ba319edf99218f3188d79a145701d8c8724adf9a/spark/src/main/scala/org/apache/comet/serde/CometHighOrderFunction.scala#L195). I reproduced a wrong result through full Spark/Comet execution. With one Parquet row containing `a=[0,1]`: ```sql SELECT filter(a, x -> x = 0 OR monotonically_increasing_id() = 0) FROM t; ``` Spark and JVM dispatch return **`[0,1]`**; native execution returns **`[0]`**. Evaluating the skipped right-hand side advances the counter. The same guard also misses: - Guarded `abs(INT_MIN)`: native throws with ANSI enabled. - Guarded `element_at(..., 0)`: native throws with ANSI disabled. - Guarded `rand(42L)`: native selects different elements. - Reusing an optimized ANSI Dataset after disabling session ANSI: guarded division throws natively. All five cases match Spark when native HOF execution is disabled. Preserve per-element evaluation masks, or conservatively dispatch affected bodies using their captured error modes. The native build, JVM compilation, and **all four SQL fixture tests passed**. Additional end-to-end checks confirmed the empty-array fix, guarded-division control, captures, shadowing, and three-level nesting. [Current CI](https://github.com/apache/datafusion-comet/actions/runs/35438214183) still awaits approval; the full Spark-version matrix was not run. Nothing posted to GitHub. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
