andygrove opened a new pull request, #4578: URL: https://github.com/apache/datafusion-comet/pull/4578
## Which issue does this PR close? Closes #4560. ## Rationale for this change `array_size(NULL)` should return `NULL`, but Comet returned `-1`. `array_size` is `RuntimeReplaceable` and lowers to `Size(child, legacySizeOfNull = false)`. With `legacySizeOfNull = false`, Spark returns `NULL` for a `NULL` input; with `legacySizeOfNull = true` it returns `-1`. Comet's `Size` serde built the `else` branch of its `CASE WHEN` from the session conf `SQLConf.get.legacySizeOfNull` rather than from the expression's own `legacySizeOfNull` field. Since `spark.sql.legacy.sizeOfNull` defaults to `true`, the `else` literal became `-1` even though `array_size` had explicitly set the expression flag to `false`. The query runs natively via `CometProject`, so it did not fall back: it just returned the wrong value. ## What changes are included in this PR? `CometSize.convert` now reads `expr.legacySizeOfNull` instead of `SQLConf.get.legacySizeOfNull`. This honors the flag baked into each `Size` instance, which already accounts for both the `array_size` override (always `false`) and the ANSI handling Spark applies when constructing the expression. The regular `size` and `cardinality` paths are unaffected because their expressions carry the resolved session value. ## How are these changes tested? Added a test in `CometArrayExpressionSuite` asserting `array_size` returns `NULL` for a `NULL` array input under both `spark.sql.legacy.sizeOfNull=false` and `true`. The existing `size - respect to legacySizeOfNull` test continues to pass, confirming the `size` path is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
