marvelshan opened a new pull request, #4580:
URL: https://github.com/apache/datafusion-comet/pull/4580
## Which issue does this PR close?
Closes #4472
## Rationale for this change
Spark's `Size` expression accepts both `ArrayType` and `MapType` inputs, but
Comet previously marked `MapType` as `Unsupported`, causing `size()` on maps to
always fall back to Spark. The native Rust implementation already had
`DataType::Map` handling in `spark_size_array`; the only missing pieces were
the `ScalarValue::Map` arm in `spark_size_scalar` and the serde gate that
blocked MapType from reaching native execution.
## What changes are included in this PR?
- Added `ScalarValue::Map` arm to `spark_size_scalar` in
`native/spark-expr/src/array_funcs/size.rs` so the scalar code path also
handles map inputs
- Changed `CometSize.getSupportLevel` from `Unsupported` to `Compatible()`
for `MapType` in `spark/src/main/scala/org/apache/comet/serde/arrays.scala`
- Removed `getUnsupportedReasons` override (MapType is no longer
unsupported) and the outdated reason string
- Enabled `test_spark_size_map_array` Rust test (removed `#[ignore]`, fixed
`MapArray::try_new` API usage)
- Changed `size.sql` SQL test from `query spark_answer_only` to `query` so
`size(m)` runs through Comet
- Updated `posexplode.sql` fallback reason to match the actual serde reason
from Explode/GenerateExec
- Promoted `ignore("size with map input")` to `test("size with map input")`
in `CometMapExpressionSuite`, removed the now-invalid fallback test
## How are these changes tested?
- Rust unit test: `test_spark_size_map_array` verifies map size calculation
(2 entries, 1 entry, empty map, null map returning -1)
```bash
cd native && cargo test --package datafusion-comet-spark-expr
test_spark_size_map_array
```
- SQL file test: `size.sql` now exercises `size(m)` through Comet native path
```bash
./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite size"
-Dtest=none
```
- ScalaTest: `CometMapExpressionSuite` "size with map input" uses column
references to avoid constant folding
```bash
./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometMapExpressionSuite"
```
- posexplode fallback reason update:
```bash
./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite posexplode"
-Dtest=none
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
