viirya commented on code in PR #6726:
URL: https://github.com/apache/datafusion-comet/pull/6726#discussion_r4244147318
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometSparkToColumnarExec.scala:
##########
@@ -138,13 +138,20 @@ object CometSparkToColumnarExec extends
CometSink[SparkPlan] with DataTypeSuppor
CometScanWrapper(nativeOp, CometSparkToColumnarExec(op))
}
+ /**
+ * Declines calendar intervals at any depth. Arrow holds the time part of an
interval in
+ * nanoseconds, so `IntervalMonthDayNanoWriter` overflows on one of more
than about 292 years,
+ * which Spark accepts (#5279).
+ */
override def isTypeSupported(
dt: DataType,
name: String,
fallbackReasons: ListBuffer[String]): Boolean = dt match {
- case ArrayType(StringType, _) => true
- case MapType(StringType, StringType, _) => true
- case _: ArrayType | _: MapType => false
+ case CalendarIntervalType =>
Review Comment:
I can reproduce this on `48aeec051`, but it isn't specific to this PR. With
the same setup (AQE and broadcast off, 10 shuffle partitions, `DECIMAL(38,0)`
keys `0..99`, the right side an RDD with a `NullType` column on the JVM
columnar shuffle), the join returns 11 of 100 rows in each of these:
- Left is an RDD with `(decimal, int)`, which the conversion took before
this PR.
- Left is an RDD with `(decimal, array<string>)`, which `branch-1.1` already
converts.
- No conversion at all. Left is a native Parquet scan, with
`spark.comet.convert.rdd.enabled=false`.
So this is #3079. Any native shuffle hashing a wide-decimal key next to a
JVM columnar one loses rows. A guard on the conversion paths would not cover
the last case, while #6005 and #6548 fix it at the shuffle.
This PR does widen what reaches the bug. Array- and map-bearing inputs used
to stay on the columnar shuffle on both sides of such a join. @andygrove could
this land after #6005 or #6548? Once it's rebased on one of them, a join like
sunchao's, with an `array<int>` column on the converted side, would be a good
regression test to add here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
