gruuya commented on issue #16622:
URL: https://github.com/apache/datafusion/issues/16622#issuecomment-5714715902
I noticed the idea of DF LTS versions being discussed in the context of the
DF 56.0.0 release. As someone who's gone through many many DF updates (we
haven't missed a single one since DF 9.0.0!), I'd like to share my comments.
> This means for an application that relies on multiple downstream packages
must wait until ALL of them have upgraded to the new version in order to
upgrade DataFusion. If there is any delay in the downstream libraries updating,
it delays.
I see this as the original motivating factor behind this, and I know the
pain. However, as someone already mentioned, I think this is the best use-case
for AI tools, which greatly diminish, if not outright remove all the friction
associated with this process. Thus I'd question whether DataFusion even needs
to commit to `less frequent major / breaking releases` in the first place (in
this day and age)?
Arguably, AI tools might not be evenly distributed yet, and human review can
still be a point of friction in the upgrade process, so the premise (`less
frequent major / breaking releases`) might still be justifiable. In that case
I'd favor option 1 rather than LTS branches, as I fear LTS branches shift some
of the maintenance burden to consumer crates (delta-rs/iceberg-rust/etc.),
which might not have the resources/discipline to properly handle it, and so it
would end up being a waste of effort on the DF community part.
To be concrete, consider the following options the maintainers of the DF
consumer crates have:
1. (status quo) ignore the DF LTS versions, and just keep updating the main
branch with the latest DF version when released
- LTS effort wasted
2. (wrong approach) start pinning the main branch to DF LTS versions
- defer the upgrade effort to later (when it's much bigger)
- doesn't really resolve the original problem, since it now blocks the
user from picking up newer DF versions since some crates are stuck on LTS
3. ("right" approach) maintain separate LTS branches of their own, that
track DF LTS branches
- more maintenance work than 1 or 2, they not only have to manage these
branches/releases, but also likely do some backporting of their own
fixes/features from their main branch
- what about their main branch now?
- If it just tracks DF LTS, then there's no point in a separate LTS
branch to begin with, and we're back to case 2
- but then again if it doesn't then it may as well track the newest
DF version, and we're back to case 1 (thus defeating the purpose of a separate
LTS branch and rendering the DF LTS effort wasted again)
- defer the upgrade effort (of the LTS branch) to later (when it's much
bigger)
None of these options actually benefit the end user and the crate
maintainers simultaneously, so I'm wondering what would be the canonical way of
using DF LTS versions then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
