andygrove commented on PR #5051: URL: https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5122851449
> Tried out on a pipeline that relies heavily on caching and got an error: > > ``` > org.apache.spark.SparkException: Comet execution only takes Arrow Arrays, but got class org.apache.spark.sql.execution.vectorized.OffHeapColumnVector. This typically happens when a Comet scan falls back to Spark due to unsupported data types (e.g., complex types like structs, arrays, or maps). To resolve this, you can: (1) enable spark.comet.scan.allowIncompatible=true to use a compatible native scan variant, or (2) enable spark.comet.convert.parquet.enabled=true to convert Spark Parquet data to Arrow format automatically. > at org.apache.spark.sql.comet.util.Utils$.$anonfun$getBatchFieldVectors$1(Utils.scala:417) > at org.apache.spark.sql.comet.util.Utils$.$anonfun$getBatchFieldVectors$1$adapted(Utils.scala:403) > at scala.collection.immutable.Range.map(Range.scala:60) > at org.apache.spark.sql.comet.util.Utils$.getBatchFieldVectors(Utils.scala:403) > at org.apache.spark.sql.comet.util.Utils$.$anonfun$serializeBatches$1(Utils.scala:248) > ``` > > Note that I've tried to enable the suggested config insertions, but it didn't work either. For context: pipeline has lots of fallbacks to default Spark operators. Also using Iceberg V2 as datasource with MOR tables. > > cc @andygrove Thanks @sandugood. I'm looking into it now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
