Improving Backward Compatibility

[Collin Anderson]

Hi All,

I think it would help if Django was better at *deprecations and backward 
compatibility*. I’ve brought this up [before 
], 
but didn’t get any feedback, so here’s another try at it, with some *specific 
ideas of how to improve things below*.

*Background*:

The [API docs ] 
say “*If, for some reason, an API declared stable must be removed or 
replaced*, it will be declared deprecated but will remain in the API for at 
least two feature releases.” - I’ve always thought of the “must be removed” 
as “there’s really no alternative”. I think that if it's not broken, Django 
should avoid breaking it.

I maintain lots of Django projects, and upgrading is not a small task. 
There are enough changes to undocumented APIs already happening that it 
would be great if the documented ones didn't change as much.

I also think it’s a little disappointing that *3rd party libraries don’t 
get easy 1.8 and 2.0 support out of the box*. Both of those releases are 
currently supported, but if the library uses deprecated features to support 
1.8, it won’t work on 2.0 by definition.

Yes, backward compatibility means Django has more technical debt, and yes 
it comes at a cost (more code, more docs, tests run longer, more thank-less 
maintenance etc), but it means Django’s not forcing users to do the tedious 
work of changing their code. That’s a _huge_ benefit. Isn’t the whole point 
of a framework to have documented and tested code so the user doesn’t need 
to do tedious things? “it takes care of much of the hassle of Web 
development, so you can focus on writing your app.”

Some deprecations I thought could have gone better: #17209 (auth views), 
#26013 (urls.urlresolvers), #22218 (url patterns()), #23276 (url() 
strings), #14675 (urls.defaults), #6735 (views.generic.simple), #18651 
(assignment_tag).

*Ideas of how to improve backward compatibility*:

*1.* I think it would help if any breaking change, and any new 
deprecation/removal (any change that adds to those sections in the release 
notes) received 3-7 days *feedback from the mailing list*, with a clear 
subject, something like “deprecating X”, "changing x's defaults", “renaming 
X” or “removing X”. It's a little more bureaucracy, but I think there's a 
huge benefit to making sure everyone's on the same page about these things. 
(This already happens for some, but not all deprecations.)

I don’t pay close attention to tickets and pull requests, so I often don’t 
notice that there’s a new deprecation/removal until _after_ it’s already 
been merged. The few people on the ticket may have agreed to removing 
something, but not necessarily the larger community. Or, if it was being 
discussed, a lot of discussion was on the new feature, and it wasn’t clear 
that a deprecation/removal would be included in the change. (Sometimes the 
deprecation is thrown in as an after-thought / "while we're at it, lets 
rename...")

Having a *clear email in django-developers* would also give more people the 
initial heads-up to avoid using the old code.

*2.* For some changes, Django has decided to delay/*hold off 
deprecating/removing a feature until after the next LTS or “eventually”* 
(some good examples: #23433 (django-admin.py), #27753 (utils.six, etc), 
#25236 (ifequal), #25978 (render_to_response), #28593 (url() -> re_path()). 
I think this delay is a really good thing, and I think it should be *more 
official/documented to encourage its use* (if not the default). As [Russ 
put it 

]:

“The only other suggestion I've got is to add a new "pre-deprecation" step 
to our deprecation process - a flag that lets us indicate that at some 
point, we intend to deprecate something, but we haven't decided when that 
will be. This would essentially be a "*don't use this on new code, but 
there's no rush in replacing it*". It could also be accompanied with a 
Warning so that existing uses could be found and replaced if someone was so 
inclined.”

Delaying the removal would *help libraries support both 1.8 and 2.0* (for 
example) at the same time. (Also, I think a longer removal timeline might 
make it a little less likely for something to be deprecated/removed in the 
first place. - A good thing, in my opinion.)

Thanks,
Collin

[before] 
https://groups.google.com/d/msg/django-developers/ZWy2Esj46nE/jzSP3DRIEAAJ

[API docs] https://docs.djangoproject.com/en/2.0/misc/api-stability/

[Russ put it] 
https://groups.google.com/d/msg/django-developers/J7vpMKSHk7U/3vGPs71MCgAJ

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-de

Re: Improving Backward Compatibility

[Tim Graham]

I don't see a strong reason to make it easy to support 1.8 and 2.0 at the 
same time. Support for Django 1.8 ends 4 months after the release of 2.0. 
As the 2.0 release notes say, "Following the release of Django 2.0, we 
suggest that third-party app authors drop support for all versions of 
Django prior to 1.11. At that time, you should be able to run your 
package’s tests using python -Wd so that deprecation warnings do appear. 
After making the deprecation warning fixes, your app should be compatible 
with Django 2.0."

Third-party packages aren't required to support a certain version of Django 
until it's completely unsupported. In fact, a package dropping support for 
a version of Django that's nearing it's end-of-life is a good reminder for 
users to upgrade their Django.

In the past few weeks, I sent a few pull requests to various third-party 
packages following the "drop support for versions older than 1.11" 
guideline and I thought this process was very clean. I'm not enthusiastic 
about the idea of making the deprecation process even more lengthy. I think 
it will mostly promote the continued use of deprecation features resulting 
in messier code and encourage prolonged use of unsupported versions of 
Django.

Personally, I think asking developers to do a bit of house cleaning every 
three years if you want to upgrade from LTS to then next LTS is reasonable. 
What's your experience been like?

On Wednesday, December 6, 2017 at 4:12:00 PM UTC-5, Collin Anderson wrote:
>
> Hi All,
>
> I think it would help if Django was better at *deprecations and backward 
> compatibility*. I’ve brought this up [before 
> ],
>  
> but didn’t get any feedback, so here’s another try at it, with some *specific 
> ideas of how to improve things below*.
>
> *Background*:
>
> The [API docs ] 
> say “*If, for some reason, an API declared stable must be removed or 
> replaced*, it will be declared deprecated but will remain in the API for 
> at least two feature releases.” - I’ve always thought of the “must be 
> removed” as “there’s really no alternative”. I think that if it's not 
> broken, Django should avoid breaking it.
>
> I maintain lots of Django projects, and upgrading is not a small task. 
> There are enough changes to undocumented APIs already happening that it 
> would be great if the documented ones didn't change as much.
>
> I also think it’s a little disappointing that *3rd party libraries don’t 
> get easy 1.8 and 2.0 support out of the box*. Both of those releases are 
> currently supported, but if the library uses deprecated features to support 
> 1.8, it won’t work on 2.0 by definition.
>
> Yes, backward compatibility means Django has more technical debt, and yes 
> it comes at a cost (more code, more docs, tests run longer, more thank-less 
> maintenance etc), but it means Django’s not forcing users to do the tedious 
> work of changing their code. That’s a _huge_ benefit. Isn’t the whole point 
> of a framework to have documented and tested code so the user doesn’t need 
> to do tedious things? “it takes care of much of the hassle of Web 
> development, so you can focus on writing your app.”
>
> Some deprecations I thought could have gone better: #17209 (auth views), 
> #26013 (urls.urlresolvers), #22218 (url patterns()), #23276 (url() 
> strings), #14675 (urls.defaults), #6735 (views.generic.simple), #18651 
> (assignment_tag).
>
> *Ideas of how to improve backward compatibility*:
>
> *1.* I think it would help if any breaking change, and any new 
> deprecation/removal (any change that adds to those sections in the release 
> notes) received 3-7 days *feedback from the mailing list*, with a clear 
> subject, something like “deprecating X”, "changing x's defaults", “renaming 
> X” or “removing X”. It's a little more bureaucracy, but I think there's a 
> huge benefit to making sure everyone's on the same page about these things. 
> (This already happens for some, but not all deprecations.)
>
> I don’t pay close attention to tickets and pull requests, so I often don’t 
> notice that there’s a new deprecation/removal until _after_ it’s already 
> been merged. The few people on the ticket may have agreed to removing 
> something, but not necessarily the larger community. Or, if it was being 
> discussed, a lot of discussion was on the new feature, and it wasn’t clear 
> that a deprecation/removal would be included in the change. (Sometimes the 
> deprecation is thrown in as an after-thought / "while we're at it, lets 
> rename...")
>
> Having a *clear email in django-developers* would also give more people 
> the initial heads-up to avoid using the old code.
>
> *2.* For some changes, Django has decided to delay/*hold off 
> deprecating/removing a feature until after the next LTS or “eventually”* 
> (some good examples: #23433 (django-admin.py), #27753 (utils.six, etc), 

Re: Improving Backward Compatibility

[Josh Smeaton]

Following on from Collin, another ticket that generated a large amount of 
code churn for little perceived benefit 
was https://code.djangoproject.com/ticket/21127 (enforce on_delete). I know 
there are many in the community that rail against any and all changes (I 
had an argument with such a person on the 2.0 reddit thread recently), but 
we should endeavour to make churn as painless and infrequent as possible. 
As a concrete example with the on_delete ticket, we could have used 
settings to control the default on_delete behaviour per nullable and 
non-nullable foreign key. I know we don't like adding extra settings, and 
perhaps another path forward could have sufficed, but requiring everyone to 
update every foreign key definition is quite hostile.

In that light, I think a thread per deprecation on django-dev is a great 
idea, which will both help to keep us honest, and give those in the 
community that strongly dislike backward incompatible changes a chance to 
propose alternatives.

I'm not so keen on the idea of keeping deprecations going for as long as 
Collin suggests. There should be little harm in 3rd party libraries 
declaring **this is the final version to support 1.8**, and then bumping 
the major version to support the next major Django series. The new LTS to 
LTS policy that Django has taken up is new - but should give 3rd party 
libraries a bit of certainty on how to support Django moving forward.

The only concern I have is 3rd party libraries might begin to stop 
supporting versions of X.0 and X.1, and instead just support the LTS (X.2). 
That'll mean fewer users to run and test newer versions in production, if 
3rd party ecosystem isn't keeping current. I don't know if that'll be an 
actual problem in practise though. I don't think it will be.

For renames and moves, have we considered leaving around aliases 
indefinitely? How big is the overhead when the modules themselves are kept? 
We could generate an `ObsoleteWarning` + undocument to discourage usage. I 
don't wish to see or maintain large or complicated deprecations. At the 
same time, dropping renames seems like overkill. Could we maintain a 
catalog of renames in a single spot, then have the startup machinary patch 
old names? At least we'd avoid polluting the "real code" with shims.

On Thursday, 7 December 2017 09:08:09 UTC+11, Tim Graham wrote:
>
> I don't see a strong reason to make it easy to support 1.8 and 2.0 at the 
> same time. Support for Django 1.8 ends 4 months after the release of 2.0. 
> As the 2.0 release notes say, "Following the release of Django 2.0, we 
> suggest that third-party app authors drop support for all versions of 
> Django prior to 1.11. At that time, you should be able to run your 
> package’s tests using python -Wd so that deprecation warnings do appear. 
> After making the deprecation warning fixes, your app should be compatible 
> with Django 2.0."
>
> Third-party packages aren't required to support a certain version of 
> Django until it's completely unsupported. In fact, a package dropping 
> support for a version of Django that's nearing it's end-of-life is a good 
> reminder for users to upgrade their Django.
>
> In the past few weeks, I sent a few pull requests to various third-party 
> packages following the "drop support for versions older than 1.11" 
> guideline and I thought this process was very clean. I'm not enthusiastic 
> about the idea of making the deprecation process even more lengthy. I think 
> it will mostly promote the continued use of deprecation features resulting 
> in messier code and encourage prolonged use of unsupported versions of 
> Django.
>
> Personally, I think asking developers to do a bit of house cleaning every 
> three years if you want to upgrade from LTS to then next LTS is reasonable. 
> What's your experience been like?
>
> On Wednesday, December 6, 2017 at 4:12:00 PM UTC-5, Collin Anderson wrote:
>>
>> Hi All,
>>
>> I think it would help if Django was better at *deprecations and backward 
>> compatibility*. I’ve brought this up [before 
>> ],
>>  
>> but didn’t get any feedback, so here’s another try at it, with some 
>> *specific 
>> ideas of how to improve things below*.
>>
>> *Background*:
>>
>> The [API docs ] 
>> say “*If, for some reason, an API declared stable must be removed or 
>> replaced*, it will be declared deprecated but will remain in the API for 
>> at least two feature releases.” - I’ve always thought of the “must be 
>> removed” as “there’s really no alternative”. I think that if it's not 
>> broken, Django should avoid breaking it.
>>
>> I maintain lots of Django projects, and upgrading is not a small task. 
>> There are enough changes to undocumented APIs already happening that it 
>> would be great if the documented ones didn't change as much.
>>
>> I also think it’s a little disappointing that *3rd

Re: Improving Backward Compatibility

[Aymeric Augustin]

Hello,

The duration of the deprecation period doesn't make a big difference because 
few people update their code until they have no other choice. Keeping 
deprecated code around longer means it's harder to track down the release where 
the corresponding changes were introduced and figure out what changes are 
needed. This effect already occurs in Django 2.0 where some removals are 
related to 1.9 and others to 1.10.

Requiring a thread of django-developers for deprecations sounds interesting. 20 
to 40 features were deprecated in the recent releases. That's one feature per 
week or fortnight. We'd likely deprecate fewer features if we had to reach 
consensus on django-developers every time.

Best regards,

-- 
Aymeric.



> On 7 Dec 2017, at 04:09, Josh Smeaton  wrote:
> 
> Following on from Collin, another ticket that generated a large amount of 
> code churn for little perceived benefit was 
> https://code.djangoproject.com/ticket/21127 
>  (enforce on_delete). I know 
> there are many in the community that rail against any and all changes (I had 
> an argument with such a person on the 2.0 reddit thread recently), but we 
> should endeavour to make churn as painless and infrequent as possible. As a 
> concrete example with the on_delete ticket, we could have used settings to 
> control the default on_delete behaviour per nullable and non-nullable foreign 
> key. I know we don't like adding extra settings, and perhaps another path 
> forward could have sufficed, but requiring everyone to update every foreign 
> key definition is quite hostile.
> 
> In that light, I think a thread per deprecation on django-dev is a great 
> idea, which will both help to keep us honest, and give those in the community 
> that strongly dislike backward incompatible changes a chance to propose 
> alternatives.
> 
> I'm not so keen on the idea of keeping deprecations going for as long as 
> Collin suggests. There should be little harm in 3rd party libraries declaring 
> **this is the final version to support 1.8**, and then bumping the major 
> version to support the next major Django series. The new LTS to LTS policy 
> that Django has taken up is new - but should give 3rd party libraries a bit 
> of certainty on how to support Django moving forward.
> 
> The only concern I have is 3rd party libraries might begin to stop supporting 
> versions of X.0 and X.1, and instead just support the LTS (X.2). That'll mean 
> fewer users to run and test newer versions in production, if 3rd party 
> ecosystem isn't keeping current. I don't know if that'll be an actual problem 
> in practise though. I don't think it will be.
> 
> For renames and moves, have we considered leaving around aliases 
> indefinitely? How big is the overhead when the modules themselves are kept? 
> We could generate an `ObsoleteWarning` + undocument to discourage usage. I 
> don't wish to see or maintain large or complicated deprecations. At the same 
> time, dropping renames seems like overkill. Could we maintain a catalog of 
> renames in a single spot, then have the startup machinary patch old names? At 
> least we'd avoid polluting the "real code" with shims.
> 
> On Thursday, 7 December 2017 09:08:09 UTC+11, Tim Graham wrote:
> I don't see a strong reason to make it easy to support 1.8 and 2.0 at the 
> same time. Support for Django 1.8 ends 4 months after the release of 2.0. As 
> the 2.0 release notes say, "Following the release of Django 2.0, we suggest 
> that third-party app authors drop support for all versions of Django prior to 
> 1.11. At that time, you should be able to run your package’s tests using 
> python -Wdso that deprecation warnings do appear. After making the 
> deprecation warning fixes, your app should be compatible with Django 2.0."
> 
> Third-party packages aren't required to support a certain version of Django 
> until it's completely unsupported. In fact, a package dropping support for a 
> version of Django that's nearing it's end-of-life is a good reminder for 
> users to upgrade their Django.
> 
> In the past few weeks, I sent a few pull requests to various third-party 
> packages following the "drop support for versions older than 1.11" guideline 
> and I thought this process was very clean. I'm not enthusiastic about the 
> idea of making the deprecation process even more lengthy. I think it will 
> mostly promote the continued use of deprecation features resulting in messier 
> code and encourage prolonged use of unsupported versions of Django.
> 
> Personally, I think asking developers to do a bit of house cleaning every 
> three years if you want to upgrade from LTS to then next LTS is reasonable. 
> What's your experience been like?
> 
> On Wednesday, December 6, 2017 at 4:12:00 PM UTC-5, Collin Anderson wrote:
> Hi All,
> 
> I think it would help if Django was better at deprecations and backward 
> compatibility. I’ve brought this up [before 
> 

Re: Improving Backward Compatibility

[Josh Smeaton]

> We'd likely deprecate fewer features if we had to reach consensus on
django-developers every time

I’m not sure if you consider this a good thing or a bad thing Aymeric :)

I haven’t fully thought through a process but I’m not even sure a full
consensus would be required. But it’d help to think through options that
may not have been considered on the ticket which is a smaller audience.

Perhaps a “we’re proposing deprecating this way, unless others have a
better idea how we can proceed” would be enough?
On Thu, 7 Dec 2017 at 18:12, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Hello,
>
> The duration of the deprecation period doesn't make a big difference
> because few people update their code until they have no other choice.
> Keeping deprecated code around longer means it's harder to track down the
> release where the corresponding changes were introduced and figure out what
> changes are needed. This effect already occurs in Django 2.0 where some
> removals are related to 1.9 and others to 1.10.
>
> Requiring a thread of django-developers for deprecations sounds
> interesting. 20 to 40 features were deprecated in the recent releases.
> That's one feature per week or fortnight. We'd likely deprecate fewer
> features if we had to reach consensus on django-developers every time.
>
> Best regards,
>
> --
> Aymeric.
>
>
>
> On 7 Dec 2017, at 04:09, Josh Smeaton  wrote:
>
> Following on from Collin, another ticket that generated a large amount of
> code churn for little perceived benefit was
> https://code.djangoproject.com/ticket/21127 (enforce on_delete). I know
> there are many in the community that rail against any and all changes (I
> had an argument with such a person on the 2.0 reddit thread recently), but
> we should endeavour to make churn as painless and infrequent as possible.
> As a concrete example with the on_delete ticket, we could have used
> settings to control the default on_delete behaviour per nullable and
> non-nullable foreign key. I know we don't like adding extra settings, and
> perhaps another path forward could have sufficed, but requiring everyone to
> update every foreign key definition is quite hostile.
>
> In that light, I think a thread per deprecation on django-dev is a great
> idea, which will both help to keep us honest, and give those in the
> community that strongly dislike backward incompatible changes a chance to
> propose alternatives.
>
> I'm not so keen on the idea of keeping deprecations going for as long as
> Collin suggests. There should be little harm in 3rd party libraries
> declaring **this is the final version to support 1.8**, and then bumping
> the major version to support the next major Django series. The new LTS to
> LTS policy that Django has taken up is new - but should give 3rd party
> libraries a bit of certainty on how to support Django moving forward.
>
> The only concern I have is 3rd party libraries might begin to stop
> supporting versions of X.0 and X.1, and instead just support the LTS (X.2).
> That'll mean fewer users to run and test newer versions in production, if
> 3rd party ecosystem isn't keeping current. I don't know if that'll be an
> actual problem in practise though. I don't think it will be.
>
> For renames and moves, have we considered leaving around aliases
> indefinitely? How big is the overhead when the modules themselves are kept?
> We could generate an `ObsoleteWarning` + undocument to discourage usage. I
> don't wish to see or maintain large or complicated deprecations. At the
> same time, dropping renames seems like overkill. Could we maintain a
> catalog of renames in a single spot, then have the startup machinary patch
> old names? At least we'd avoid polluting the "real code" with shims.
>
>
> On Thursday, 7 December 2017 09:08:09 UTC+11, Tim Graham wrote:
>
> I don't see a strong reason to make it easy to support 1.8 and 2.0 at the
>> same time. Support for Django 1.8 ends 4 months after the release of 2.0.
>> As the 2.0 release notes say, "Following the release of Django 2.0, we
>> suggest that third-party app authors drop support for all versions of
>> Django prior to 1.11. At that time, you should be able to run your
>> package’s tests using python -Wdso that deprecation warnings do appear.
>> After making the deprecation warning fixes, your app should be compatible
>> with Django 2.0."
>>
>
>>
>> Third-party packages aren't required to support a certain version of
>> Django until it's completely unsupported. In fact, a package dropping
>> support for a version of Django that's nearing it's end-of-life is a good
>> reminder for users to upgrade their Django.
>>
>> In the past few weeks, I sent a few pull requests to various third-party
>> packages following the "drop support for versions older than 1.11"
>> guideline and I thought this process was very clean. I'm not enthusiastic
>> about the idea of making the deprecation process even more lengthy. I think
>> it will mostly promote the continued use