andygrove opened a new pull request, #100: URL: https://github.com/apache/datafusion-java/pull/100
## Which issue does this PR close? <!-- No tracking issue; small, self-contained enhancement. --> - N/A ## Rationale for this change DataFusion publishes Apache Spark–compatible functions and expression planners in the [`datafusion-spark`](https://crates.io/crates/datafusion-spark) crate, but `datafusion-java` had no way to use them. This adds an opt-in so Java callers can build a `SessionContext` whose SQL surface matches Spark semantics (e.g. `crc32`), without affecting the default context. ## What changes are included in this PR? - **Java:** a new `SessionContextBuilder.withSparkFunctions()` setter, carried through the existing `SessionOptions` protobuf path. It composes with all other builder options; the no-arg `new SessionContext()` path is unchanged (Spark functions are strictly opt-in). - **Proto:** a new `optional bool spark_functions` field on `SessionOptions`. - **Native:** a `build_session_context` helper that, when the flag is set, builds the context through `SessionStateBuilder::with_default_features().with_spark_features()` (so Spark implementations override built-ins of the same name). Object-store registration and the memory tracker are preserved. - **Cargo:** the `datafusion-spark` (v53.1.0, matching the pinned DataFusion) dependency behind a new optional `spark` feature, included in the default feature set — mirroring the existing object-store feature gating. Builds that strip the feature surface a clear error from the JVM if the flag is requested. - **Docs:** a "Spark-compatible functions" section in the SessionContext user guide. ## Are these changes tested? Yes. - Java: proto round-trip tests for the flag, plus an integration test asserting `crc32('Spark')` returns `1557323817` with the flag and that the same SQL fails without it (proving the flag is load-bearing). - Rust: unit tests gated on the `spark` feature confirming `crc32` is registered when the flag is set and absent otherwise. - Full suites pass: Rust `cargo test --features spark`, JVM `./mvnw -pl core test` (334 tests), and `cargo clippy --features spark -- -D warnings` clean. Verified the native crate also compiles with `--no-default-features`. ## Are there any user-facing changes? Yes — a new public `SessionContextBuilder.withSparkFunctions()` method and accompanying user-guide documentation. No breaking changes to existing APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
