I'm almost exclusively a lurker on this list, but a constant user, and have in the past a keen observer in the datetime discussions.
I think you've made your case well. I'd be happy for this migration from Django to be as aggressive as the maintainers are comfortable. I agree that doing step 1 at a minimum seems like the right thing to me. I'm very interested in using zoneinfo over pytz for my projects, and I'm willing to shoulder more risk in order to do so. I hope Django will allow me to make that choice. On Wed, Jun 17, 2020 at 9:32 AM Paul Ganssle <p...@ganssle.io> wrote: > Greetings all, > > Now that PEP 495 <https://www.python.org/dev/peps/pep-0495/> (the fold > attribute, added in Python 3.6) and PEP 615 > <https://www.python.org/dev/peps/pep-0615/> (the zoneinfo module, added > in Python 3.9) have been accepted, there's not much reason to continue > using pytz, and its non-standard interface is a major source of bugs > <https://blog.ganssle.io/articles/2018/03/pytz-fastest-footgun.html>. In > fact, the creator and maintainer of pytz said during the PEP discussions > that he was looking forward to deprecating pytz > <https://discuss.python.org/t/pep-615-support-for-the-iana-time-zone-database-in-the-standard-library/3468/16>. > After merging zoneinfo upstream into CPython, I turned the reference > implementation into a backport to Python 3.6+ > <https://github.com/pganssle/zoneinfo>, which I continue to maintain > alongside the upstream zoneinfo module, so it is already possible to > migrate Django *today*. > > Right now, Django continues to use pytz as its source of time zones, even > for simple zones like UTC; I would like to propose starting the migration > to zoneinfo *now*, to enable Django users to start doing their own > migrations, and so there can be ample time for proper deprecation cycles. I > apologize in advance for the huge wall of text, the TL;DR summary is: > > - It's important to start this migration as soon as possible, because > it will take a long time and some of the bugs it fixes get worse as time > goes on; right now Django's assumption that all time zones are from pytz > will likely block Django users from migrating before Django does. > - The combination of pytz's non-standard interface and Hyrum's Law > <https://www.hyrumslaw.com/> makes this less straightforward than one > would hope. Likely the best solution is to adopt "zoneinfo" immediately > with a wrapper that deprecates the pytz-specific interface. I have provided > a library to do this <https://pypi.org/project/pytz-deprecation-shim/>. > - There are some open questions (which may only be open because I > don't know Django's process well enough) in the migration plan: > - Should this change be put under a feature flag of some sort? If > so, what should be the default values for coming releases? > - Should *all* the pytz-specific tests be kept with zoneinfo tests > added, or should they be migrated and pytz tests confined to a smaller > subset? > > *Rationale* > > Before I get into the detailed migration plan, I'd like to lay out the > case for *why* this needs to be done. I expect it to be at least somewhat > painful, but and I'd prefer it if y'all were convinced that it's the right > thing to do *before* you hear about the costs 😉. I am not the kind of > person who thinks that just because something is in the standard library it > is automatically "better" than what's available on PyPI, but in the case of > pytz, I've been recommending people move away from it from a long time, > because most people don't know how to use it correctly. The issue is that > pytz was originally designed to work around issues with ambiguous and > imaginary times that were fixed by PEP 495 in Python 3.6, and the > compromise it made for correctness is that it is not really compatible with > the standard tzinfo interface. That's why you aren't supposed to directly > attach a timezone with the tzinfo argument, and why datetime arithmetic > requires normalization. Any substantial code base I've ever seen that uses > pytz either has bugs related to this or had a bunch of bugs related to this > until someone noticed the problem and did some sort of large-scale change > to fix all the bugs, then imposed some sort of linting rules. > > In addition to the "pytz is hard to use" problems (which, honestly, should > probably be enough), pytz also has a few additional issues that the > maintainer has said will not be fixed (unless pytz becomes a thin wrapper > around zoneinfo, which is at best just using a slower version of zoneinfo). > The biggest issue, which is going to become increasingly relevant in the > coming years, is that it only supports the Version 1 spec in TZif files, > which (among other issues), does not support datetimes after 2038 (or > before 1902) — this was deprecated 15 years ago, and it is unlikely that > you will find modern TZif files that don't have Version 2+ data. Pytz also > does not support sub-minute offsets, which is mostly relevant only in > historical time zones. And, of course, pytz is not compatible with PEP 495 > and in some ways really cannot be made compatible with PEP 495 (at least > not easily). > > Of course, one reasonable objection here is that Django is a huge install > base with very long release cycles, and it doesn't make sense for Django to > be an experimental "early adopter" of the new library. This is a reasonable > response, but it's actually *because* it has a huge install base and long > release cycles that it's important for Django to migrate early, because it > can't "turn on a dime" like that. The long release cycles mean that changes > made *now* won't be universal for many years, and it's important for > users to have a long notice period that change is coming (particularly > since, as time goes on, Year 2038 bugs will become more and more common). > The huge install base means that *at a minimum*, zoneinfo should be > *supported*, to allow users to start their own migrations today. > > *Migration Plan* > > I am fairly certain this is going to be a tricky migration and will > inevitably come with *some* user pain. I don't think this will be Python > 2 → 3 style pain, but some users who have been doing the "right thing" with > pytz will need to make changes to their code in the long run, which is > unfortunate. I think there are several stages of "support" for zoneinfo, > not all of which are mandatory. > > 1. Drop all requirements that time zones support pytz-specific > interfaces internally — make it so that end users have the *option* to > use zoneinfo and datetime.timezone. > 2. Document that users *should* use zoneinfo, even if it's not the > default. > 3. Convert all uses of pytz to using a shim that supports pytz's > interface, but raises warnings whenever something pytz-specific is used. > [0] I have already created a library for this purpose > <https://pypi.org/project/pytz-deprecation-shim/>. [1] This has three > sub-stages: > 1. Make this optional, *enabled* by a feature flag. [2] > 2. Make this optional, *disabled* by a feature flag. > 3. Remove all feature flags related to this, make it mandatory. > 4. Remove the shims, making zoneinfo the default for all options, but > maintaining support for user-supplied pytz zones (again this can be rolled > out under feature flags if desired). > 5. Remove support for pytz entirely. > > I am not sufficiently familiar with Django's development process to know > if feature flags are frequently used. I recommend *at a minimum* doing #1 > immediately and adding tests for compatibility with the zoneinfo module. > Preferably that would be accompanied with #2 as well. > > I have created a Proof of Concept PR > <https://github.com/django/django/pull/13073> using the deprecation shims > (#3), but with no feature flags, which I think is the *fastest* you > should move on this. It was a fairly minimal change, which was encouraging, > but a great many of the tests still have an explicit dependency on pytz. > This *will break some users*, as is probably evident from the fact that I > needed to make changes *other than* the constructors and fixes to > warnings. Encouragingly, the only test I needed to touch was to fix a > warning, not an error; to the extent that the Django test suite describes > Django's public interface, this is a "non-breaking change", but as I > mention in the pytz-deprecation-shim migration guide > <https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html>, > there are some semantic differences that you could easily encounter if you > are counting on `django.utils.timezone.get_current_timezone()` returning a > pytz time zone instead of a shim class. The majority of people won't > encounter these, but... Hyrum's Law. > > One remaining question here is what to do with the pytz-specific tests. > Until Django gets to stage 5 (pytz zones aren't even *supported*), there > should be at least *some* tests for pytz support, but presumably most of > the tests that explicitly use pytz are only doing so incidentally (e.g. > almost all uses of `pytz.utc`). In stage 3, I would think that most > pytz-specific tests should either be parameterized to test with both pytz > and zoneinfo or tested only with zoneinfo. > > I am not terribly familiar with the release schedule or backwards > compatibility guarantees that Django makes in point versions, etc. I read > the Django documentation on stability, but it suffers from the same > problems that SemVer in general suffers from (and that there's no avoiding, > really), which is that breaking changes are in the eye of the beholder. I > leave it to y'all to decide the roll-out schedule for this stuff (assuming > it's accepted at all), but I'm happy to offer what advice I can on the > matter. > > For those of you who have made it this far, thank you for indulging me. > > Best, > Paul > > > [0] There's an additional step between 2 and 3 that can be taken, which is > the adoption of pytz-deprecation-shim (or an equivalent thereof), but only > for UTC. Most (all?) of the semantic differences between pytz and the shims > don't come up with fixed-offset zones or UTC, and a shim around UTC to > provide localize and normalize + warning can be a wrapper around > datetime.timezone.utc rather than the much newer zoneinfo. > > [1] I created pytz-deprecation-shim specifically to help migrate big code > bases and popular libraries like Django and pandas off pytz, but if you are > concerned with taking on an additional dependency, it's not terribly > difficult to extract the core of the library directly into Django (for > example, Django doesn't need any of the stuff for Python 2 compatibility), > but the downside there is that what support exists for a Django fork would > lag behind the upstream library. > > [2] Unless this is a build-time flag, there's no way to have Django > defaulting to having pytz as a required dependency with an option to not > depend on it. > > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/b6e8e75c-dddf-da65-3af5-43f1b5b23eca%40ganssle.io > <https://groups.google.com/d/msgid/django-developers/b6e8e75c-dddf-da65-3af5-43f1b5b23eca%40ganssle.io?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABpHFHT%2BZ0Kwcvx3aT-dHwoA-hfgAP-S%2BkyJBXkchaRgASHDuQ%40mail.gmail.com.
