sandugood commented on PR #5051:
URL:
https://github.com/apache/datafusion-comet/pull/5051#issuecomment-5208500029
Tested on Spark4.0 against current branch. Using PySpark
Got this error on the same pipeline that was testing earlier. And I can
clearly see a regression in performance. Same job with `.cache()` on default
Spark took <10 min, Comet ran >15min.
Here is error:
```
Py4JJavaError: An error occurred while calling o12641.createOrReplace.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 92
in stage 27.0 failed 4 times, most recent failure: Lost task 92.3 in stage 27.0
(TID 15624) (1.1.1.1 executor 76): org.apache.comet.CometNativeException:
Invalid argument error: column types must match schema types, expected
List(Struct("col_1": Utf8, "col_2": Boolean, "col_3": Boolean)) but found
List(Struct("col_1": Utf8, "col_2": non-null Boolean, "col_3": Boolean)) at
column index 5
at org.apache.comet.Native.executePlan(Native Method)
at
org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:155)
at
org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:154)
at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:189)
at
org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:154)
at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
at
org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:152)
at
org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:196)
```
(obfuscated column names)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
