Smallfu666 opened a new pull request, #6179:
URL: https://github.com/apache/datafusion-comet/pull/6179
## Which issue does this PR close?
Closes #5995.
Stacked on #6178, which has to land first. Until it merges this branch also
shows its commit.
## Rationale for this change
`TRY_CAST` between map types whose key cast can fail fails in Comet with
`Found unmasked nulls for non-nullable StructArray field "key"`, where Spark
returns a result.
`CometCast.isSupported` decided map support by recursing into the key and
value casts, so the
plan was marked `Compatible`.
## What changes are included in this PR?
- Report `Unsupported` for a TRY map cast unless its key cast is on the list
of casts that cannot
throw, the same list #6178 adds for the dispatcher. Arrow's map format
requires a non-nullable
key field, so no Comet path can hold the result. Spark's own result is not
consistent between
readers either: its map holds a null key internally, `map_keys` shows it,
and `collect`,
`element_at` and a cast to string read it as `0`.
- The codegen dispatcher then refuses the tree, through #6178, and the plan
falls back to Spark.
- Widening key casts, and plain `CAST` in legacy and ANSI mode, stay native.
Without #6178, `Unsupported` would route the cast through the dispatcher,
which wrote the null
key as `0` and turned today's error into a wrong result.
One known limitation, which is effectively unreachable. The struct arm of
`isSupported` returns
the first field that is not `Compatible`, so a struct whose earlier field is
`Incompatible` would
report `Incompatible` even when a later field is this map cast. The only
`Incompatible` cast today
is a negative-scale decimal to string with the legacy flag off, and such a
decimal cannot be
created in that state. Even then it only matters with
`spark.comet.expr.allowIncompatible=true`.
By default an `Incompatible` cast goes to the codegen dispatcher, which
refuses the tree through
#6178 and falls back. With the flag set it would run natively and fail with
the error above
rather than return a wrong result.
## How are these changes tested?
- `try_cast_map_null_key.sql`: the direct `try_cast`, `map_keys` and
`element_at` over a narrowing
key cast, and `map_keys` over a date to timestamp key cast, all fall back.
A legacy `CAST` and a
widening TRY cast stay native, asserted with `expect_native`. The direct
cast fails on #6178
alone with the error above.
- A unit test in `CometNativeCastSuite` pins the support level across eval
modes, for a date key,
and for a map nested in an array and in a map value.
- `CometNativeCastSuite` and `CometCodegenSuite` pass, and `test-compile`
passes on the spark-3.4,
3.5, 4.0, 4.1 and 4.2 profiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
