andygrove opened a new issue, #325:
URL: https://github.com/apache/datafusion-comet/issues/325
### What is the problem the feature request solves?
### What is the problem the feature request solves?
We currently delegate to DataFusion when casting from string to decimal and
there are some differences in behavior compared to Spark.
- An input of `4e7` produces `40000000.00` in Spark, and `null` in DataFusion
- Inputs of `.`, `-`, `+` and empty string produce `null` in Spark, and
`0.0` in DataFusion
- Input of `0` produces `0` in Spark, and `null` in DataFusion
- Arrow-rs does not support negative scale (`Cannot cast string to decimal
with negative scale`). We could choose to fallback to Spark for this use case
(or if `SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED` is enabled)
### Describe the potential solution
_No response_
### Additional context
I used the following test in `CometCastSuite` to explore this.
```scala
test("cast string to decimal") {
val values = generateStrings(numericPattern, 5).toDF("a")
castTest(values, DataTypes.createDecimalType(10, 2))
castTest(values, DataTypes.createDecimalType(10, 0))
withSQLConf((SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key,
"true")) {
castTest(values, DataTypes.createDecimalType(10, -2))
}
}
```
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
