Re: Ideas for a new DEP on long-term Django API compatibility

[Aymeric Augustin]

Hello Pakal,

If I understand correctly, you're proposing that:

1. The Django project should maintain a higher level of backwards compatibility.
2. This would be easier to achieve outside the main codebase, in a submodule or 
in a separate repository.
3. This would reduce the amount of work required for maintaining Django 
(triaging tickets, fixing bugs, adding features).

Backwards-compatibility is a major concern for the development of Django, as 
you can see from in contribution documentation, in the archives of this mailing 
list, in DEPs, etc. Maintaining more compatibility with the same effort or less 
should be uncontroversial. Now the question is to get consensus that your 
proposal will indeed achieve the stated goals.

I'm quite skeptical of the "same effort or less". The more behaviors Django 
maintains — and you're proposing essentially to maintain all behaviors that 
existed at all previous releases — the more complexity accrues. You can't solve 
that just by shuffling deprecation warnings or compatibility imports in another 
module (and it seems to me that it would be harder to maintain, for the reason 
the Python ecosystem usually eschews monkey-patching: non local effects). 
Keeping old ways around forever increases the barrier to mastering Django for 
people who haven't been writing Django code for ten years like you and me. It 
also increases the scope on which compatibility must be maintained, and that 
can get in the way of making Django better.

One way to frame the debate is: does Django aim at being a fossilizing 
ecosystem or a living one? Up to this point, we've chosen to maintain a living 
ecosystem. At a given point in time, all maintained libraries support 2 or 3 
versions of Django (latest, previous, LTS). Things change a bit; unmaintained 
libraries that don't adapt fall out of favor; new libraries build upon the 
experience of previous ones and try to do better. One can see this as progress 
and a way to keep up with the moving web ecosystem. One can also see this as 
useless churn. As you call it "walk or die", it's pretty clear which camp 
you're in :-) I'd say there's truth in both!

As an experiment, let's just consider the first backwards-incompatible change 
from the latest release: 
https://docs.djangoproject.com/en/2.2/releases/2.2/#admin-actions-are-no-longer-collected-from-base-modeladmin-classes
 

 (I'm skipping the database backends changes because they're explicitly out of 
the backwards-compatibility policy, but we started documenting them for the 
maintainers of third-party database backends.) This is a good example. Django 
behaved in a non-Pythonic way; it didn't respect the Principle of Least 
Astonishment. Since we believe Django is alive and will have infinitely more 
future users than past users, we make the change. How can we maintain 
backwards-compatibility for this? We can try to detect if subclasses have all 
the actions of their parent classes and, if not, raise a warning. But then we 
raise inappropriate deprecations warnings in legitimate use cases where a 
subclass mustn't have some actions of its parent class, which is probably the 
use case for which this change was requested.

Experimenting with a third party module like you did is absolutely the way to 
go. You're already ahead of the usual advice :-) Keep in mind that the barrier 
for making a third-party project an official Django project or to merging it in 
Django is very high. With 1 fork and 2 stars, django-compat-patcher doesn't 
pass it yet.

Writing a DEP to formalize arguments on both sides is a valid idea. Don't 
forget the other side. In your freecodecamp article, you're spending two lines 
on the downsides of Compat Patchers, and this is not enough. How would Django 
communicate to users which backwards-incompatible changes are covered by Compat 
Patchers and which aren't? How would we respond to users asking for a Compat 
Patcher for a backwards-incompatible change for which it's impossible? 
Eventually, will every library document the list of Compat Patchers it requires 
or it's incompatible with?

As you can see, there's more than "blinding self-absorption" and "harmful 
psychological bias" here. Your freecodecamp article makes valid points. It also 
misses completely what's hard in maintaining backwards compatibility. I wish 
you hadn't found it useful to insult the work put into managing 
backwards-compatibility over the years and the people who did it.

Best regards,

-- 
Aymeric.



> On 29 Jun 2019, at 13:09, Pkl  wrote:
> 
> Hello everyone,
> 
> I'm planning on writing a PEP on long-term API stability for Django.
> Most of the rationale behind this kind of commitment is described here : 
> https://www.freecodecamp.org/news/api-stability-is-cheap-and-easy/
> 
> The idea is that the django ecosystem has suffered a lot, over the past 
> decade, from the 

Re: Discussing improvements of django.setup()

[Aymeric Augustin]

Hello Pakal,

Making django.setup() idempotent

In https://code.djangoproject.com/ticket/28752 
, you say:

> I've been bitten numerous times by the impredictable behaviour of django when 
> django.setup() was called numerous times.
> In the old days I had exceptions, now it's mainly subtle breakages of logging 
> configuration.

This is fuzzy for a bug report. Ideally, you should say:

- What you're doing (minimal reproduction instructions, within supported use of 
Django)
- What results you're expecting
- What results you're getting

Anyway it suggests that the problem is about logging. In this comment 
 I 
said that the only possible bug is that `configure_logging()` isn't idempotent. 
Your answer 
 confirms 
that the issue is really about logging. Let's fix configure_logging(), then!

Barring new arguments in favor of the proposed approach, which involves nested 
locks, I'm -1. Nested locks create a possibility of deadlocks. See my comment 
on the PR for details.

Making django.setup() tweakable

Again, the rationale is fuzzy. I'm seeing two use cases:

- pytest-django
- django-compat-patchers

One more or one less monkey patch isn't going to make a difference to these 
projects ;-)

I'm ready to accept that there's a need for projects that aren't big hacks but 
I need to see it explained clearly and specifically. I'd really like to see a 
feature request in the form of:

- I want to do X, which is a reasonable thing to do and stays within supported 
use of Django, except I need to override django.setup()
- Here's what the code looks like with a workaround (monkey-patching, probably)
- Here's what the code would look like with the improvement I'm asking for 
(hopefully much better)
- Here are the alternatives I considered and why I rejected them
- Other use cases also include Y and Z

I'm asking this because concrete use cases will put us in a better position to 
find the best solution.

Best regards,

-- 
Aymeric.



> On 27 Jun 2019, at 23:26, Pkl  wrote:
> 
> Hello everyone,
> 
> I'm bringing here, for broader review and discussion, the subject of making 
> django setup more "tweakable":
> https://code.djangoproject.com/ticket/30536
> https://github.com/django/django/pull/11435
> 
> There is indeed a need for doing pre/post operations at that crucial moment 
> of the framework lifetime, especially to apply compatibility shims 
> (greening-related, API stability-related...) and setup test scaffolding, 
> stubs etc.
> 
> A separate PR has been created to handle threadsafety/idempotence 
> (https://github.com/django/django/pull/11440) when multiple actors try to 
> setup django concurrently, but a discussion remains on how to potentially 
> allow users to specify an alternative setup 
> 
> I originally did a POC with a new django setting, but if we want to have full 
> control, even settings might not be normally loaded at that stage. So maybe, 
> as apollo13 suggested, an environment variable, like DJANGO_SETUP_CALLABLE, 
> complementing the DJANGO_SETTINGS_MODULE one?
> What are your thoughts about this ?
> 
> thanks,
> regards,
> Pakal
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/0206c98c-199f-49f9-9ff7-cba736e71224%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4673AD3D-4EDF-44FB-931F-B3E231D6311D%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Ideas for a new DEP on long-term Django API compatibility

[Florian Apolloner]

Hello World,

On Sunday, June 30, 2019 at 9:24:05 AM UTC+2, Aymeric Augustin wrote:
>
> As an experiment, let's just consider the first backwards-incompatible 
> change from the latest release: 
> https://docs.djangoproject.com/en/2.2/releases/2.2/#admin-actions-are-no-longer-collected-from-base-modeladmin-classes
>  
> 
>

I think Aymeric is making a good point here on why any project like 
django-compat-patcher is ultimately doomed to be a failure. You simply 
cannot magically "fix" changes like this without knowing user intent. I can 
also remember the change from get_query_set to get_queryset (1.6 IIRC), you 
simply cannot magically patch that to work with multiple sublcasses and 
dynamic attributes. There will __always__ be situations where this will 
break in horrible and subtle ways. I'd argue that this would be worse then 
not being able to use a project that is not updated for a while. 

Pakal wrote in his initial mail:

> With *less* work than currently, except small changes in procedures, we 
> can revive the majority of existing packages (except python2vs3 troubles), 
> ...
>

This is a bold argument for which I'd like to see some proof, also *less* 
work for whom? 

And last but not least, if one assumes all your arguments hold true, then 
why isn't django-compat-patcher used by existing 3rd party libraries (At 
least according to public usage 
https://github.com/pakal/django-compat-patcher/network/dependents)? Either 
the usecase you are suggesting isn't as strong as you make it to be or 3rd 
party packages are maintained well enough for this to be not a problem? 
Personally I also think that a package that wasn't updated since 2015 
should probably not be magically patched to theoretically work on a current 
Django.

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7b076492-0ad7-4a06-8bef-008cd6fd91f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.