wirybeaver opened a new pull request, #2731:
URL: https://github.com/apache/datafusion-ballista/pull/2731
# Which issue does this PR close?
Part of #1241; follows the build/release-boundary discussion in #2638.
# Rationale for this change
An Iceberg adapter can lag behind a Ballista DataFusion/Arrow upgrade.
Keeping that adapter in the root workspace makes its compatibility work part of
workspace-wide CI. This PR proposes the **upstream merge target** for an opt-in
Iceberg build gate, with SDK implementations built and released separately. The
personal-fork demo is supporting evidence, not a replacement for #2638.
Ballista already has codec override APIs. This proposal standardizes a small
Iceberg envelope and adapter interface using those APIs; it does not add a new
general plugin mechanism. Dependency isolation comes from the independent
workspace and release policy, not from a Cargo feature alone.
# What changes are included in this PR?
- Add the SDK-free `ballista-core/contrib-iceberg` feature. Provider and
leaf-scan adapters own serialization payloads and SDK operations; Ballista owns
versioned framing and delegation to the existing codecs, including function and
expression methods.
- Exclude `integrations/iceberg` from the root workspace and document
independent manifests, lockfiles, CI and release selection. No Iceberg
dependency or root lockfile changes.
- Add gate-only compilation to CI.
```text
client provider / scheduler scan
|
SDK-free codec gate ---- unhandled operations ---> existing codec
|
independent Iceberg adapter
(SDK, payload, catalog, reconstruction)
```
Codec delegation is adapted from #2638 at
`86bc19c46df93d707ca40bbaefe1e1cfcbc542f5`. The envelope is
`ballista-contrib-iceberg-v1\0`, distinct from that PR's experimental framing.
Integrations still need compatible DataFusion/Arrow types, and Ballista still
maintains the gate's DataFusion-facing interfaces.
# Are there any user-facing changes?
An opt-in API and integration-build guide. Enabling the feature alone does
not provide an Iceberg reader. Independently released adapters should select a
compatible released Ballista version; the demo uses path dependencies only to
exercise the proposed gate before release.
# Test
The [personal-fork
demo](https://github.com/wirybeaver/datafusion-ballista/pull/12) adapts #2638
to this gate and passes its local-catalog end-to-end scenarios: standalone
queries observe new commits while static tables and views retain their pinned
versions, and a Flight SQL client reads scheduler-mounted Iceberg catalogs
through an executor. The scan round trip also preserves snapshot, projection,
filter and fetch. Docker/S3 coverage is retained but was not run locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
