andygrove opened a new pull request, #5032: URL: https://github.com/apache/datafusion-comet/pull/5032
## Which issue does this PR close? There is no dedicated issue. Related to the expression coverage epic #240. ## Rationale for this change `pow`/`power` (`Pow`), `translate` (`StringTranslate`), and `to_csv` (`StructsToCsv`) have native implementations that are known to be incompatible with Spark (float pow correctness, grapheme vs code-point handling in translate, CSV formatting differences). Because they are incompatible by default, the serde declines to convert them, so the entire enclosing operator falls back to Spark unless the user opts in with `allowIncompatible`. The JVM codegen dispatcher runs Spark's own generated code inside the native Comet pipeline, which is bit-exact with Spark. Routing these expressions through it by default keeps the operator native while remaining fully Spark-compatible, and leaves the faster native path available via `allowIncompatible`. This is the same `NativeOptInAvailable` pattern already used by `to_json`, `from_json`, and the `split`/regex family. This change was identified by the expression coverage audit (#5031): these were the scalar entries classified as INCOMPATIBLE. The two remaining INCOMPATIBLE entries, `explode_outer` and `posexplode_outer`, are not addressed here because their incompatibility is in the `GenerateExec` operator, which codegen dispatch (an expression-level mechanism that rejects generators) cannot handle. ## What changes are included in this PR? - `CometPow`, `CometStringTranslate`, and `CometStructsToCsv` now mix in `NativeOptInAvailable` and route through the codegen dispatcher by default (`Compatible` with a native opt-in), falling to the native path only when `allowIncompatible` is set. For `to_csv`, the native path additionally keeps its existing type restrictions. - The `pow` and `translate` SQL file tests, which previously asserted `expect_fallback`, now assert native execution. A new `to_csv` SQL file test covers quoting and escaping. - Support notes for `pow`, `power`, `translate`, and `to_csv` in the expression guide. ## How are these changes tested? The Comet SQL file tests for `pow`, `translate`, and `to_csv` run each query through both Spark and Comet, verify the results match, and verify Comet executes the expression natively (through the dispatcher) rather than falling back. `string_translate_enabled.sql` continues to cover the native opt-in path. Coverage includes NaN/Infinity and zero/negative bases for `pow`, deletion and empty-argument cases for `translate`, and delimiter/quote escaping for `to_csv`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
