andygrove opened a new issue, #24146: URL: https://github.com/apache/datafusion/issues/24146
### Is your feature request related to a problem or challenge? `datafusion-cli` does not register the `datafusion-spark` function library, so there is no interactive way to run a Spark-compatible function. It fails as though the function does not exist: ``` $ ./target/debug/datafusion-cli -c "SELECT next_day(arrow_cast(95026236,'Date32'),'Mon');" Error during planning: Invalid function 'next_day'. Did you mean 'today'? ``` The only way to execute one today is to add a query to a file under `datafusion/sqllogictest/test_files/spark/` and run the sqllogictest target. That works, but it is a poor fit for exploration: every probe is an edit plus a test run, throwaway queries have to be deleted afterwards, and the error above sends people looking for a missing implementation rather than a missing registration. `datafusion-cli/Cargo.toml` has no `datafusion-spark` dependency at all today. ### Describe the solution you'd like Make the Spark function library available in `datafusion-cli` behind an explicit opt-in. A CLI flag such as `--spark`, a `SET` option, or both. ### Describe alternatives you've considered **Registering unconditionally is not viable.** `datafusion_spark::register_all` overrides any existing function with the same name, and `SessionStateBuilderSpark::with_spark_features` documents the same behavior: > Note: This overwrites any previously registered items with the same name. Names present in both libraries include `abs`, which behaves differently: `datafusion-spark`'s `abs` wraps on integral overflow when `datafusion.execution.enable_ansi_mode` is false, while core `abs` raises. A default-on registration would silently change results for every existing CLI user with no indication in the query that a different function ran. The opt-in has to be explicit for that reason, not merely for tidiness. An `EXPLAIN`-visible or banner indication of which library is active would help here, since the shadowing is otherwise invisible. ### Additional context Raised in review of https://github.com/apache/datafusion/pull/23893, where the CLI's behavior had to be documented as a pitfall for the `audit-datafusion-spark-expression` skill. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
