peterxcli commented on code in PR #5457:
URL: https://github.com/apache/datafusion-comet/pull/5457#discussion_r3848549958
##########
spark/src/main/scala/org/apache/comet/SparkErrorConverter.scala:
##########
@@ -117,8 +117,13 @@ object SparkErrorConverter extends ShimSparkErrorConverter
{
val summary: String = errorJson.summary.getOrElse("")
- // Delegate to version-specific shim - let conversion exceptions propagate
- val optEx = convertErrorType(errorJson.errorType, errorClass, params,
sparkContext, summary)
+ // Math.multiplyExact throws a plain JVM exception in every Spark version,
without an
+ // ANSI error class or configuration advice. Delegate other errors to the
version-specific shim.
+ val optEx = if (errorJson.errorType == "LongOverflow") {
Review Comment:
also handle `IntegerOverflow` here? as I remember there are at least these
two type of arithmetic overflow in spark sql error result
##########
native/spark-expr/src/conversion_funcs/cast.rs:
##########
@@ -418,6 +419,36 @@ pub(crate) fn cast_array(
Ok(spark_cast_postprocess(cast_result?, &from_type, to_type))
}
+/// Recursive casts must not evaluate child values hidden by a null parent or
outside a slice.
+fn prepare_nested_cast_input(array: ArrayRef) -> DataFusionResult<ArrayRef> {
Review Comment:
looks like this is for:
> Checked casts must also ignore values that are not logically present. A
native `IF(flag, s, NULL)` can leave an overflowing date in the child buffer of
a null struct or map. Casting that null container must return null, not
evaluate the hidden date and throw. Sliced lists and maps can likewise retain
unused child values before or after their visible rows.
so it seems like the root cause is native `IF(flag, s, NULL)` doesnt
actually respect the null buffer? should we turn to fix it instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
