andygrove commented on issue #3920: URL: https://github.com/apache/datafusion-comet/issues/3920#issuecomment-4297164752
Four tests are currently skipped with `assume(!isSpark40Plus)` for this issue: - `CometExpressionSuite`: `to_json` - `CometExpressionSuite`: `to_json escaping of field names and string values` - `CometExpressionSuite`: `to_json unicode` - `CometJsonExpressionSuite`: `to_json - all supported types` All four fail on Spark 4 with the same fallback: `COMET: invoke is not supported`, and the plan retains a regular `Project` instead of native execution. Root cause: `StructsToJson` in Spark 4 extends `RuntimeReplaceable` and its `replacement` is `Invoke(Literal.create(evaluator, ObjectType(classOf[StructsToJsonEvaluator])), "evaluate", ...)`. Comet's serde sees the post-replacement `Invoke` (pretty name `"invoke"`), not the original `StructsToJson`. `CometStaticInvoke` handles `StaticInvoke` only; there is no serde for `Invoke`. Fix direction: add an `Invoke` handler that recognizes `Invoke(Literal(_: StructsToJsonEvaluator, _), "evaluate", _, _)` and routes to `CometStructsToJson`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
