wirybeaver commented on PR #2731: URL: https://github.com/apache/datafusion-ballista/pull/2731#issuecomment-6563643858
@milenkovicm, here is the tradeoff I am trying to highlight: | Tradeoff | #2638: SDK in root workspace | #2731: SDK-free gate + independent integration | |---|---|---| | Core maintenance | No new core API | Small interface and codec wrapper | | Dependency maintenance | SDK graph enters the shared lockfile | SDK graph stays with the integration | | Upgrade lag | Can fail required workspace-wide CI | Integration can retain its compatible Ballista version | | Release coordination | Maintainers must accommodate adapter compatibility | Integration upgrades and releases independently | For a concrete example, suppose Ballista’s next release is ready: its core crates pass testing against the upgraded DataFusion/Arrow stack, but the pinned Iceberg adapter does not compile against it. With Iceberg included in required workspace-wide checks, Ballista maintainers must either: - Wait for the upstream adapter to catch up. - Patch or fork the adapter, update dependency pins, and rerun integration tests. - Exclude the integration from required checks and document the resulting support gap. Each option adds work during release preparation. With an independently released integration, Ballista can release after validating its core and gate interface, while the Iceberg integration continues supporting its previous compatible Ballista version. My concern is asking the same limited group of maintainers to resolve an external SDK’s compatibility gap while preparing a Ballista release. I expect that ongoing coordination to cost substantially more than maintaining a small, SDK-free, feature-gated interface. The dependency isolation comes from the independent workspace and release policy; the feature flag alone would not provide it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
