dwsmith1983 opened a new issue, #7083:
URL: https://github.com/apache/datafusion-comet/issues/7083
## Describe the bug
`CometMapFromEntries` routes some inputs through the JVM codegen dispatcher
(a `FLOAT` or `DOUBLE` key, a `BinaryType` key or value, or
`spark.sql.mapKeyDedupPolicy=LAST_WIN`), but it does not check the result type
first. When the map's values are calendar intervals whose elapsed time is
larger than about 292 years, the dispatcher's output writer converts
microseconds to the nanoseconds of Arrow's `IntervalMonthDayNano` with
`Math.multiplyExact`, and the query aborts with `ArithmeticException: long
overflow`. Spark returns the map.
This is the overflow tracked in #5279, reached through `map_from_entries`.
Found in #5867, which makes `map_from_arrays` decline the dispatcher for a
calendar-interval result (so it falls back to Spark) but leaves
`map_from_entries` as it is on `main`.
## Steps to reproduce
On `main` (f180d65f1), Spark 4.1, default configuration:
```sql
CREATE TABLE repro_mfe_interval(s decimal(18, 6)) USING parquet;
INSERT INTO repro_mfe_interval VALUES (10000000000.000000), (1.000000),
(NULL);
SELECT map_from_entries(array(named_struct('k', 1.0D, 'v', make_interval(0,
0, 0, 0, 0, 0, s))))
FROM repro_mfe_interval;
```
Comet fails the task:
```
java.lang.ArithmeticException: long overflow
at java.base/java.lang.Math.multiplyExact(Math.java:1004)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificCometBatchKernel.process(Unknown
Source)
at
org.apache.comet.udf.codegen.CometScalaUDFCodegen.evaluate(CometScalaUDFCodegen.scala:160)
at org.apache.comet.udf.CometUdfBridge.evaluate(CometUdfBridge.java:150)
```
The same query restricted to `s = 1` matches Spark.
## Expected behavior
Until #5279 carries microseconds through the dispatcher losslessly,
`CometMapFromEntries` should decline the dispatcher when its result holds a
calendar interval, so the expression falls back to Spark, the same way
`map_from_arrays` does in #5867 (`dispatchDeclineReason` returning
`CalendarIntervalOutput.declineReason(expr)`), with a SQL test for the
`map_from_entries` case.
## Additional context
`map_from_entries` has a single argument, so the shared-subexpression
decline that `map_from_arrays` also uses is not needed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
