andygrove commented on PR #5714: URL: https://github.com/apache/datafusion-comet/pull/5714#issuecomment-5970550332
Closing this in favor of #6564. Since #6542, the codegen dispatcher only calls into Spark's own classes, so the fused tree this PR builds now falls back to Spark. I confirmed it by merging main locally: 10 of the 18 tests in `CometTypedDatasetSuite` fail with `invoke calls scala.Function1, which is not part of Spark`. Getting it working again would take an exemption to #6542's policy plus special cases for decimal serializers. #6564 takes a simpler route. It leaves the typed operation on Spark and converts the output of `SerializeFromObjectExec` to Arrow, so the operators above it can run natively. Measured with this PR's benchmark, it got 93-100% of the fused speedup wherever either approach paid off, and it covers `flatMap`, `mapPartitions`, `mapGroups` and `cogroup` as well as `map`. A trimmed version of the benchmark moved over with it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
