Hello everyone,
(apologies in advance for the long post, but there are lots of aspects to dig here, it's not for the pleasure of writing but a necessary to progress on the relevance of a DEP) *"As you can see, there's more than "blinding self-absorption" and "harmful psychological bias" here. Your freecodecamp article makes valid points. It also misses completely what's hard in maintaining backwards compatibility. I wish you hadn't found it useful to insult the work put into managing backwards-compatibility over the years and the people who did it."* I'd rather address that immediately: as you read in my article, "blinding self-absorption", "harmful psychological bias" and other strong words were all aimed at something very specific, the "feeling of return to purity" that a person could have when destroying compatibility shims. I persist and sign regarding this one: ruining the day of thousands of anonymous plugin/website maintainers all over the world should *always* be made under compulsion of heavy constraints, and with regrets; never lightheartedly. Now, maybe my attempt at explaining the roots of nowadays' mass breakages by a cultural problem are all wrong (my bad), and there are other reasons behind the *apparently* reckless compatibility destructions that I witness left and right. But what are they? Since it's not technical inability, since it's not laziness, since it's not ideology, what makes that (as far as I'm concerned) the Django ecosystem fails on that aspect, contrary to loads of others "big" frameworks, runtimes, libraries etc? If it's "we don't have enough resources", it's all understandable, but at least let's make it clear. The weirdest, in the whole story, is the global idea that everything is fine, that compatibility is already a "major concern" and treated so, that maybe it can be improved a little bit but no need to push it. It's not that way I see it. And a problem can't be addressed if it's not first recognized as so. Now, I know that it requires a good amount of diplomacy to criticize a behaviour, or here a whole policy, without concerned people feeling attacked. I don't have this kind of talent (and probably won't ever), so all I can do is be honest with my view on the situation, and bring attempts at solutions. It's precisely because Django is a great piece of software (with its omnipresent lazy-objects, it's automatic admin and model forms, its migration system, its focus on security headers...) that having it scarifying itself is unbearable. So I apologize in advance for all the criticisms that expand below, but without them there'd be no need for any DEP or change in the first place. I also want to make this clear: I didn't mean that breaking changes were never justified, just that they had to be the last option, and thoroughly explained. When the AngularJS team, after the years of hype, admitted that the whole designed was flawed, reset the whole framework as Angular version 2, and made great efforts to ease part-by-part migration towards the new behaviour, I felt unlucky but not angry. When trivial changes ruined tons of pluggable Django apps that did one thing and did it well, it was a different matter. Before answering remarks, I think it's worth summarizing what the "problem" is. This is not lightly that I used the words "dependency hell". My experience with Django, purely regarding compatibility, has been abysmal compared to lots of other ecosystems (in C/C++, PHP, Jquery...) that I have crossed, and which didn't have the malleability of Python; I see several cumulated reasons explaining that : - The pace of breakages: an arm's long list of new breaking changes introduced every 8 months, dropping at the same time all shims older than 2-3 years, is imo a too fast breakage pace, even by the Web's standard. When changes are mainly aesthetical, compatibility shims had better remain for very long periods of time, else the benefit/harm balance is just indefensible. - More importantly, the *scope* of breakages: as long as it's only about fixing corner cases of the framework, and only a tiny portion of users is impacted (those who abused counter-intuitive and unpythonic behaviours), one can understand, and just hope not to be shot by the next upgrade; but here, we're dealing with changes that deliberately break about every existing application, even those which scrupulously followed official tutorials: ForeignKey's "on_delete" argument killed most migration files when becoming mandatory, moving url-related primitives left and right was as destructive as expected... - The growing "less batteries included" philosophy. Valueable contrib modules were outsourced from the codebase, and other pluggable applications (likes CMS apps) follow this trend, putting all valuable features into external plugins. This "minimal core" philosophy would work fine if main applications counter-balanced it with a more delicate approach to breaking changes; but it is the contrary that happens, surprisingly. So in such fast-breaking environment, only monolithic applications can avoid turning into nightmares. - The apparent needlessness of lots of changes, aggravated by the severe lack of explaining in release notes. As Linus said "And those reasons really need to be very good, and spelled out and explained". For sure I'm thankful that these release notes exist at first. I really am. Especially because without them, Django would be doomer than doomed. But even after digging the commits regarding almost 30 changes, and tracking some of them in tickets, most of them continue to look injustifiable to me. I have probably missed some incredibly deep pondering and reasonings backing them, but this miss is a problem in itself. When a perfectly suitable utility gets dropped in favor of a castrated stdlib one without a word of justification, when URL patterns (the very starting points of most pluggable apps) get broken 10 times in 5 years, it feels just like being spat on, "deal with it" style. To illustrate, I'd like to redraw the history of said URL resolving and reversal system, as I've lived it (temporal order of changes not guaranteed). - A few years years ago, you had to use the patterns() function, with tuple arguments including dotted strings, and an optional string prefix. - Then you had to provide individual url() objects instead of tuples. - Then the "url defaults", that everyone "star-imported" from as per official tutorials, changed their location. - Then patterns() were replaced by a list of urls, or by a 2-tuple (!!) when including the app_name with it. - Then url reversal by dotted strings was made impossible, one had to use named urls - Then providing urls as dotted strings became impossible, views had to be imported and inserted directly. - Then include() didn't accept a 3-tuple anymore, for some reason. - As a side effect, include(admin.site.urls) was dropped, one had to directly use admin.site.urls in patterns - Then url() got replaced by re_path(), when introducing simpler path matchers, and basic classes like RegexUrlPattern got removed. - And recently lots of utilities have been continuously transferred from django.core.urls and django.core.urlresolvers to django.urls, breaking imports on their way, for a reason that most end users ignore. My honest opinion is that, for a framework which is so reknowned and massively used, and for a set of features that are so central to every pluggable app, this avalanches of breaking changes is rather hard to sustain; especially considered the short lifespan granted to compatibility shims. Django is not just a website engine, but also a toolkit for building bigger blocks, more specialized frameworks, which themselves ought to get their own plugins, themes, bridges to interesting utilities... but it's hard to build big castles on too shaky grounds. The worse is, the forward-evolutivity of the whole system has LOWERED in the process. URL Patterns() was a powerful factory design-pattern, which could return any object it wanted; this object could fake being a list, could customize its __add__() behaviour, could accept more arguments... And what do we get in the end ? A *tuple* ([patterns], app_name), maybe the most inevolutive, semantic-less container type in Python. How can I explain all this to a Django newbie who would want to upgrade his site after 5 years of stagnation? I fear that as breaking changes become more and more normal, less and less care will be taken to pick the most evolutive data types and designs along the road. Whereas long term compatibility also requires a habit of enforcing powerful scaffolding (like forcing user code to inherit from some base clases and their metaclasses), so that these can step in at any moment and fix future changes transparently. The Way of the Cross that has been the python2to3 migration taught us that the pace of core developers is far from the pace of the silent mass of users (python3.0 was released more than 10 years ago, and I still get panicked requests of enterprises trying to migrate their huge codebase of python and C extensions); that there is always big optimism bias when introducing breaking changes (remember the u"" notation, that had to be reintroduced back because its removal was too much burden for project maintainers?); that preferring purity to practicality is always a bad idea (it seemed ugly to leave import aliases in the stdlib, as a result all big projects import from *six*, so it's the same ugliness and twice the porting efforts). I feel that the whole Django ecosystem needs more awareness on this point. Another example is the Turbogears framework. It was like a pythonist's dream, mixing "best of breeds" libraries; but whenI tried to install it, years ago, after several hours of efforts I couldn't find a "known working set". Lots of small libraries broke compatibility as if they were alone in the world, ruining (for me, YMMV) a perfectly valid concept. *"If I understand correctly, you're proposing that:1. The Django project should maintain a higher level of backwards compatibility.2. This would be easier to achieve outside the main codebase, in a submodule or in a separate repository.3. This would reduce the amount of work required for maintaining Django (triaging tickets, fixing bugs, adding features)."* Indeed. Note that just letting comptibility shims 2 or 3 times longer would also solve the problem, without resorting to new concepts. *"I'm quite skeptical of the "same effort or less". The more behaviors Django maintains — and you're proposing essentially to maintain all behaviors that existed at all previous releases — the more complexity accrues. You can't solve that just by shuffling deprecation warnings or compatibility imports in another module (and it seems to me that it would be harder to maintain, for the reason the Python ecosystem usually eschews monkey-patching: non local effects). Keeping old ways around forever increases the barrier to mastering Django for people who haven't been writing Django code for ten years like you and me. It also increases the scope on which compatibility must be maintained, and that can get in the way of making Django better."* My own experience, although much smaller than that of Django maintainers, differs quite much from these assertions. When I mentored a dozen students in internships, from a very basic knowledge of python and the web to an intermediate level in Django, I can't think of a single time where having some deprecated aliases and behaviours around hindered them. Beginners who delve into official (and up-to-date) tutorials are not supposed to meet legacy examples. If they integrate/audit third-party packages, and encounter legacy code, they might have some questions. With long-term compatibility, these questions will be answered on launch by explicit logging and warnings; with current deprecation policy, the scaffolding being forcibly destroyed, beginners will face completely abstruse crashes, or worse, will wonder for hours why their own code is never called. That's what I consider a steep learning curve (and a big waste of time). Regarding the "scope" of maintenance, of course keeping more compatibility is harder than a "fire and forget" approach. But the benefit is huge for the community, my experiments show that fixers are quite easy to maintain (only once did I have to refactor an older fixer to account for a new breaking change on the same feature), and they don't seem to block the evolution of Django at all, since they "time travel". If a blocking case occurs one day, THIS might be a valid reason to drop compatibility on a particular feature, but at leats it'll have been justified (and will have lasted as long as possible). No one demands eternal compatibility, just a breakage pace compatible with little-maintained projects and big software. As of today I can't envision how we could have Wordpress-size ecosystems, with themes shops and auto-install plugins, without it falling immediately into dependency hell. A decade seems a good start to me. *"One way to frame the debate is: does Django aim at being a fossilizing ecosystem or a living one? Up to this point, we've chosen to maintain a living ecosystem. At a given point in time, all maintained libraries support 2 or 3 versions of Django (latest, previous, LTS). Things change a bit; unmaintained libraries that don't adapt fall out of favor; new libraries build upon the experience of previous ones and try to do better. One can see this as progress and a way to keep up with the moving web ecosystem. One can also see this as useless churn. As you call it "walk or die", it's pretty clear which camp you're in :-) I'd say there's truth in both!"* I don't really buy the dilemma between "living" and "fossilizing" here. Long term compatibility doesn't encourage project maintainers to be lazy, unless new versions of the framework bring nothing interesting (which would be a problem in itself). Better compatibility just allows low-community plugins - or those who block on other little-maintained dependencies - to keep working until a next surge of activity, or until the approach of the plugin is really deemed obsolete. It encourages people to upgrade their Django instead of dreading and delaying the next upgrade. It almost makes supporting multiple versions (and LTS...) useless, since people just have to "pip -U django" if security issues arise. It removes a LOT of burden from pluggable app maintainers, time that they can allocate to handling normal tickets or - who knows - pushing their best ideas towards Django itself. What I'm sure of, is that a "living" ecosystem doesn't have to stab half his packages (and definitely murder a good part of them) at each minor version release. The awesome https://djangopackages.org/grids/ sometimes looks like a half-cemetery, in which one doesn't compare packages by features, but by "what are the odd that it still works with my particular Django version"? This hasn't to be so. "Perfectionnists with deadlines" should avoid wasting time needlessly. On the precise subject of compatibility, we should take inspiration from in-browser ecosystems like jQuery. One can take datepickers, grid displayers, notification plugins, sometimes 5 or 10 years old, they just works No need to read release nodes, to do dependency conflict resolution, to hack JS sources and rebuild, to fallback on less relevant but alive plugins. Except some rare cases (like the .live() deprecation in favor of .on()), one can just download the latest versions of big frameworks, stick them in its statifiles, and all works; in a web-browser ecosystem which is yet known for rushing like mad. Why couldn't Django offer the same thing, with all the magic Python has built-in (magic which keeps expanding, with keyword-only and soon position-only arguments, helping future compatibility shims)? In the releases notes of the last 7 versions of Django, how many compatibility shims were dropped because they really hindered Django evolution? How many trivial shims were dropped, at the contrary, "because that's what we do"? *"As an experiment, let's just consider the first backwards-incompatible change from the latest release: https://docs.djangoproject.com/en/2.2/releases/2.2/#admin-actions-are-no-longer-collected-from-base-modeladmin-classes (I'm skipping the database backends changes because they're explicitly out of the backwards-compatibility policy, but we started documenting them for the maintainers of third-party database backends.) This is a good example. Django behaved in a non-Pythonic way; it didn't respect the Principle of Least Astonishment. Since we believe Django is alive and will have infinitely more future users than past users, we make the change. How can we maintain backwards-compatibility for this? We can try to detect if subclasses have all the actions of their parent classes and, if not, raise a warning. But then we raise inappropriate deprecations warnings in legitimate use cases where a subclass mustn't have some actions of its parent class, which is probably the use case for which this change was requested."* Thanks for showcasing this one change, it illustrates perfectly lots of ideas I'm trying to push forward. This is here an example of non-trivial change; contrary to many fixers I had to code, which were just aliases and tiny wrappers - fixers asking for about zero thinking time and zero maintenance. This kind of change requires more brainstorming, but you are right on this aspect: if it's changed in-place, without any deprecation path, django-compat-patcher is not able to automatically fix it (though a fixer could expose a setting so that project maintainers indicate the ModelAdmins they want fixed). This is imho one particularly shocking and inimical change: a perfectly documented and widespread feature removed suddenly, without deprecation path if I understand correctly (nor the usual advice for lib maintainers on how to handle both behaviours), letting who knows how many Django users wondering why some controls disappeared from their website. That's a way of making people paranoid, or ensuring that the least experienced spend half their time digging on StackOverflow. Some changes are just not fixable in-place. But I know about none which cant be fixed out-of-place. In this case you mention, we should have found another suitable English word (there are 170.000+ available I heard), or combination of words, to express a similar idea. For example "admin_actions". These admin_actions attributes would have been the new documented feature, with a properly expected and pythonic behaviour. The old "actions" attribute would have silently and slowly died of old age, keeping until the end their weird but getting-stuff-done behaviour. A compatibility shim in admin submodule (or outsourced to a Compat patcher) would have handled the legacy behavior for years and decades, without harming anyone. What I'm describing here is similar to what happened to MIDDLEWARE_CLASSE=>MIDDLEWARES migration, I guess. I have no idea why these admin actions were changed in the most brutal way possible, but you are right on this point: like any compatibility shims, compatibility fixers need a tiny bit of collaboration from project maintainers, else sometimes the harm is irreparable. These "unfixable cases" are maybe the main reason I'm trying to raise awareness about compatibility (that, and the fact that I'm worried to see people waste their time in bugtrackers discussing how to work around breakages). It's one thing to be confronted to breaking changed, it's another thing to be prevented from applying one's own compatibility shims. I'd like to point that this kind of breaking chance is especially surprising as it goes against the own policy of Django : *"All the public APIs (everything in this documentation) will not be moved or renamed without providing backwards-compatible aliases. [...] If, for some reason, an API declared stable must be removed or replaced, it will be declared deprecated but will remain in the API for at least two feature releases. Warnings will be issued when the deprecated method is called. [...] We’ll only break backwards compatibility of these APIs if a bug or security hole makes it completely unavoidable."https://docs.djangoproject.com/en/2.2/misc/api-stability/https://docs.djangoproject.com/en/2.2/internals/release-process/#official-releases* *"Experimenting with a third party module like you did is absolutely the way to go. You're already ahead of the usual advice :-) Keep in mind that the barrier for making a third-party project an official Django project or to merging it in Django is very high. With 1 fork and 2 stars, django-compat-patcher doesn't pass it yet.Writing a DEP to formalize arguments on both sides is a valid idea. Don't forget the other side. In your freecodecamp article, you're spending two lines on the downsides of Compat Patchers, and this is not enough. How would Django communicate to users which backwards-incompatible changes are covered by Compat Patchers and which aren't? How would we respond to users asking for a Compat Patcher for a backwards-incompatible change for which it's impossible? Eventually, will every library document the list of Compat Patchers it requires or it's incompatible with?"* Sure I would list these remarks, but haven't most be long answered by lots of other projects? Compatibility IS maintained by default, unless its is not possible for "spelled out and explained reasons" (security, obvious contradiction of behaviour, really too heavy development effort for too few users impacted...). The good new is that people worried about compatibility could submit patches to help this effort, instead of breaking their neck against official policies. And libraries don't have to know anything about compat patchers, they are just meant to aim for the latest version of the framework, and themselves seek wide support (eg. with the django-compat lib, not to be confused with DCP), letting project maintainers tweak their patcher config as needed (by looking at Django version requirements of dependencies, or better disabling fixers' families one by one until unit-tests break, or better again checking the warnings and the usage report of the patcher - the latter not implemented yet - to know what fixers must be left enabled). I'm worried about the reversal of the burden of proof here, though. Historically, it has always been the role of people wanting to break compatibility, to justify their goal; to demonstrate that few people will be impacted; that proposed compatibility shims are wrong or unmaintainable; that the old behaviour is too harmful or blocks evolutions. Now it's as if *I* had to demonstrate (in 2019) that a strong commitment to backwards compatibility is important; that Django users and projects all around are negatively impacted by these changes; that long-term compatibility is *not* hard, nor big waste of time, nor doomed. What am I to do, pay for a world survey of Django developers? Wait for 10 years so I can tall "See? My websites are still compatible with Django 1.7, told yah"? Do big data or prophecies to compute how much time exactly maintaining compatibility shims will take to core devs? If it's a cultural axiom of the whole team that the current situation is just fine, it's not a technical issue anymore, it becomes like trying to convince a torero that corrida is bad. *"I think Aymeric is making a good point here on why any project like django-compat-patcher is ultimately doomed to be a failure. You simply cannot magically "fix" changes like this without knowing user intent. I can also remember the change from get_query_set to get_queryset (1.6 IIRC), you simply cannot magically patch that to work with multiple sublcasses and dynamic attributes. There will __always__ be situations where this will break in horrible and subtle ways. I'd argue that this would be worse then not being able to use a project that is not updated for a while."* I agree with you, in some cases, the dynamic nature of the language can play against compatibility. But once again, no one demands perfection, just a *sufficient* level of compatibility. I hadn't followed the get_query_set() removal effort (https://code.djangoproject.com/ticket/15363), I don't know what happend to the nice proposals of the ticket ("renamed_method_factory" and the likes), but all I know is that with rather limited efforts I've revived the vast majority of the modules I needed ; and upgraded to Django 2.2 my site, when major dependencies explicitely prohibited anything else than Django 1.11. That's not what I call a doomed failure ^^ *" > With *less* work than currently, except small changes in procedures, we can revive the majority of existing packages (except python2vs3 troubles), ... This is a bold argument for which I'd like to see some proof, also *less* work for whom?"* It's only a rough estimate, but I think that the time spent removing shims from the codebase is equal-or-above the rare bugfixing (or removal) of oldest compatibility fixers, for the corde dev team. Of course, I assumed the presence of deprecation paths. Labor-wise, even Compat Patchers can't beat the "Instant Death" policy that lead to the unfixable situation you both mentioned. Concerning third-party plugin maintainers, long term compatibility wouldmean a huge relief, of course, and that could "backfire" positively for everyone. *"And last but not least, if one assumes all your arguments hold true, then why isn't django-compat-patcher used by existing 3rd party libraries (At least according to public usage https://github.com/pakal/django-compat-patcher/network/dependents)? Either the usecase you are suggesting isn't as strong as you make it to be or 3rd party packages are maintained well enough for this to be not a problem? Personally I also think that a package that wasn't updated since 2015 should probably not be magically patched to theoretically work on a current Django."* Let's note that DCP is aimed at project maintainers, not library developers, who rather rely on django-compat and the likes. Without official support from Django, it'd indeed be bold of a library to impose DCP as dependency (instead of just targeting the latest Django version as it should). There is some download activity https://pypistats.org/packages/django-compat-patcher ; no idea if its bots, curious people or real users though. Concerning projects, I see lots of possible answers though: - I'm bad at (and hate doing) communication/marketing on my projects - it's a quite new concept of "companion application for long-term compatibility", so people are naturally reluctant - people naturally doubt the relevance and longevity of new packages, that are not born of figures of authority (look how asyncio overcame trio). - tons of F.U.D regarding monkey-patching and even the possibility of having long term compatibility - there is a cultural acceptation of the situation, of the resigned "necessity" to do useless forks, or waste time repairing regression with patch-ups jobs, as if it made software better - indeed people staying close of "mainstream" packages can indeed do without DCP (I just did on my latest 30-days project); but what a have fun trying to build a CMS site with usual niceties on a non-monolithic framework... I don't know if DCP is the best way to achieve long-term compatibility, I just know that it was easy to code (except the import alias mechanism), and worked like a charm for my dependency-full projects. And that when wandering on repositories I cross too many regression tickets, or even people complaining about the time it takes to upgrade (e.g. https://www.reddit.com/r/django/comments/7u84gj/django_release_schedule_and_python_3/ when I was seeking a release schedule image). I know that on this issue the interests of Django core devs are not exactly the same as those of end users, but I hope some progress can be made somehow, at least to ensure that those who need compatibility can achieve it on their own. Sorry for the long post, regards, Pascal Chambon -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6a057eb3-cd3e-4833-8318-1a43e6961fec%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
