grorge123 opened a new issue, #5525:
URL: https://github.com/apache/datafusion-comet/issues/5525
### What is the problem the feature request solves?
Any expression whose output type contains `NullType` is rejected by the
codegen dispatch gate and the whole operator falls back to Spark, even though a
`NullType` column can only ever hold nulls.
Spark's untyped constructors leave `NullType` children behind, so this hits
common literals:
- `map()` → `MapType(NullType, NullType)`
- `map('a', NULL)` → `MapType(StringType, NullType)`
- `array()` → `ArrayType(NullType)`
- `map_from_arrays(array(), array())`, `struct(map())`, and anything nested
around them
`CreateMap` and friends are routed through codegen dispatch
(`spark/src/main/scala/org/apache/comet/serde/maps.scala:214`), so a projection
containing one of these literals is currently kicked out of the native plan
with `codegen dispatch: unsupported output type ...`.
Why the gate rejects it today:
- `isSupportedDataType`
(`spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala:85`)
has no `NullType` case, so it falls through to `false`, recursively for
`ArrayType` / `StructType` / `MapType` children.
- `canHandle` applies that same predicate to both the output type
(`CometBatchKernelCodegen.scala:120`) and every `BoundReference` input
(`CometBatchKernelCodegen.scala:172`).
The rejection is only needed on the input side:
`CometScalaUDFCodegen.specFor` has no way to build an `ArrowColumnSpec` for a
`NullVector`, so a `NullType` *input* must keep falling back. On the output
side the kernel only needs to emit an all-null Arrow `NullVector`, which the
rest of the pipeline already understands (`serializeDataType` maps `NullType`
to its own type id and `Utils.toArrowField` maps it to `ArrowType.Null`).
### Describe the potential solution
Make the type gate asymmetric:
- Accept `NullType` (top-level or nested inside array / struct / map) for
the **output** type in `canHandle`, and keep rejecting it for `BoundReference`
**inputs**.
- Teach the output emitter (`CometBatchKernelCodegenOutput`) to map
`NullType` to `NullVector` and to write it with `setNull` only, without reading
a source value.
- Keep the output emitter's type surface in sync with the gate, as the
existing doc comments require, so plan-time acceptance never turns into an
execute-time exception.
- Update the Scala/Java UDF user guide: `NullType` arguments remain
unsupported, `NullType` return types become supported.
Done when a `CometSqlFileTestSuite` fixture such as `SELECT map()`, `SELECT
map('a', NULL)`, `SELECT array(map())`, and a `map()` column carried through
`ORDER BY` runs without a Spark fallback, and unit tests in
`CometCodegenSourceSuite` lock in the output/input asymmetry and the
gate/emitter agreement.
### Additional context
No native changes are needed: the `NullType` support in serde and Arrow
field conversion already exists; the missing piece is entirely in the JVM
codegen gate and output emitter.
I have a patch for this ready and will open a PR referencing this issue.
Assisted-by: Claude Code (claude-fable-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
