timsaucer opened a new pull request, #111: URL: https://github.com/apache/datafusion-java/pull/111
## Which issue does this PR close? <!-- No tracked issue yet; relates to the scan-surface discussion on upstream PRs #103 / #104. Will link an issue if one is required. --> - Closes #. ## Rationale for this change Spark users want to read DataFusion `TableProvider` data as a native Spark `DataSourceV2`. Rather than a bespoke per-operation JNI surface, this places the native boundary at a **standard ADBC driver**: Spark talks to the upstream arrow-adbc Java driver manager (`adbc-core` + `adbc-driver-jni`), which loads a native DataFusion ADBC cdylib. This reuses the upstream ADBC bindings instead of reproducing them, and keeps batches zero-copy on the cluster-provided Arrow. ## What changes are included in this PR? **Spark connector (`spark/`)** — a `DataSourceV2` registered as the `adbc-datafusion` format: - `AdbcDatafusionTableProvider`: schema probed once on the driver via `AdbcConnection.getTableSchema`. - Projection / filter / limit pushed into the scan as a Substrait plan, with a SQL fallback path. - Multi-partition reads via `executePartitioned` / `readPartition`; `target_partitions` option tunes scan parallelism. - Per-executor `AdbcConnectionPool` caches the `AdbcDatabase` per driver+options key; per-task connection (level 1) is the safe default, connection sharing (level 2) is gated off by default. - Batches imported zero-copy as `ArrowColumnVector` on the cluster-provided Arrow. **Example driver (`examples/adbc-datafusion-driver/`)** — a DataFusion ADBC cdylib exercising the full stack, with a PySpark end-to-end script and partitioning tests. **Docs** — ADBC-to-Spark partitioning guidance and connector notes. ## Are these changes tested? Yes: - Unit tests: `AdbcOptionsTest`, `AdbcConnectionPoolTest` (cache sharing + concurrency), `SubstraitPlanTest` (pushdown). - End-to-end: `AdbcSourceTest` drives Spark → arrow-adbc JNI → DataFusion ADBC cdylib → custom provider (gated on `-Dadbc.example.driver.path`). - `pyspark_e2e.py` and `tests/partitions.rs` in the example driver. ## Are there any user-facing changes? Yes — a new `adbc-datafusion` Spark data source format and its options (`driver`, `table`, `target_partitions`, ...). Additive; no changes to existing public APIs. Documentation is included. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
