On 10/9/20 2:46 PM, Kevin Henry wrote: > > It is not really possible to make the shims work the same way > because there's not enough information available to determine whether > an adjustment needs to be made. > > But since you're shimming pytz, don't you, by definition, have access > to the all the same information that it has? > No for two reasions:
1. The shims wrap zoneinfo (and dateutil, though that is not relevant in this case), they do /not/ wrap pytz (and in fact do not have a dependency on pytz, though there is some magic that allows them to play nicely with things namespaced in pytz). 2. pytz's mechanism for attaching time zones is incompatible with PEP 495. It would not really be possible to have the shims work both as pytz zones and as PEP 495 zones in all cases. You are right that it would be possible to make it so that `shim_zone.localize()` basically does what pytz does, attaching a tzinfo specific to the offset that applies at that time, and for shim_zone.normalize() to make sure that the one attached is the right one, while also allowing `tzinfo=shim_zone` to work as a PEP 495 zone. That would make it very difficult to reason about the system, though. It would make it so that depending on how you attached your time zone, you would get different semantics for comparisons and arithmetic. Sometimes normalize would work and sometimes it wouldn't. So, for example, imagine you have: def f(dt): return shim_zone.localize(dt) x = shim_zone.normalize(f(datetime(2020, 10, 31, 12) + timedelta(days=1)) If someone changes `f` to instead use `dt.replace(tzinfo=shim_zone)`, the value for `x` changes, because some function you have is no longer using `localize`. Similarly, if we have say `datetime.now(shim_zone)` return a non-localized datetime, you have differences in semantics between `shim_zone.localize(datetime.now())` and `datetime.now(shim_zone)` (both of which are valid with pytz). If we have it return a /localized/ datetime, subtraction and comparison semantics would be affected, because now times localized to one or the other offset will be inter- rather than intra-zone comparisons. Unfortunately there's simply no way to make it fully backwards and forwards compatible. The best options I see are a shim around pytz in 3.2 that just adds warnings and doesn't do anything else, followed by a hard break in 4.0 or pytz-deprecation-shim in 4.0 and hard break in 5.0. I think both are fine plans. I suspect that the slower plan will get people upgrading to 4.0 much faster, but it does have the disadvantage that some of the breakage is subtle and won't raise big errors (which is also the case, though to a lesser extent, with the faster plan). In any case, it seems uncontroversial that 3.2 should support "bring your own zoneinfo", and I think most people agree that a feature flag in 3.2 is also a good idea, so to the extent that I have time to work on this, I'll work on those things. Best, Paul > So, for example, you could wrap pytz and keep a shadow copy of the > pytz tzobject inside your tzobject, and use that to determine the > correct behavior whenever a pytz-specific call is made. So when > localize() is called you call localize on your internal object, and > store that pytz EST tzobject. Then when normalize() is called you use > that to get pytz's version of the EDT time. > > > > This occurs because localized pytz zones are different tzinfo > objects, and as such comparisons and subtraction use inter-zone semantics. > > Thank you for that example, I hadn't considered that. Unfortunately > that is another fundamental incompatibility between pytz and any > shim-around-zoneinfo (here > <https://repl.it/@severian/pytzshim-subtract> is a runnable version). > I can't think of any way around that one. > > > > Of course, there is another option, which is to, rather than > adopting a wrapper around zoneinfo, adopt a wrapper around pytz that > does not follow PEP 495, but instead just deprecates `pytz`'s API and > tells people to turn on the "use zoneinfo" feature flag. > > Agreed, that is the other main option. It has a few advantages: > > - It's backwards-compatible. > - Because it's backwards compatible it could be adopted in 3.2, > allowing a complete transition to zoneinfo by 4.2, a full two years > earlier than the shims approach. > - It only requires users to think about the change once, when they opt > in to the new approach. Using the shim means you have to think about > this issue at least twice: once when the shim is dropped in and you > have to figure out if you're affected by the backwards > incompatibilities; and once (or more) when you actually make the > change (or a series of changes) over to the native zoneinfo style. > > The main disadvantage—and a real one—is that it's more work for Django. > > > Cheers, > Kevin > On Friday, October 9, 2020 at 8:06:49 AM UTC-7 Paul Ganssle wrote: > >> Before looking at alternatives, I wonder if we can just change >> the shims package to make it fully backwards compatible? Right >> now the shims version of normalize() >> >> <https://github.com/pganssle/pytz-deprecation-shim/blob/47bd4bdd9346cafa6c6d66817082ccce099890ad/src/pytz_deprecation_shim/_impl.py#L265> >> is essentially a noop. Paul, couldn't it actually attempt to >> adjust the time the way pytz does? Perhaps by wrapping pytz >> itself, and calling its normalize() from the corresponding pytz >> timezone; or by simply replicating its time-changing logic? >> Apologies if that's a naive question. > > It is not really possible to make the shims work the same way > because there's not enough information available to determine > whether an adjustment needs to be made. The reason that > `normalize` works is that pytz attaches different `tzinfo` objects > representing fixed offsets (with a reference to the time zone they > represent) to the datetime. If arithmetic creates an invalid > datetime (i.e. a datetime in mid-June 2020 with EST attached), > `normalize` corrects this by attaching a `tzinfo` representing the > correct offset — and it does that by assuming that the UTC > datetime represented by the erroneous fixed offset is correct. > With PEP 495-style zones, you never create those datetimes with > erroneous offsets, so there's no way to tell whether a correction > is required. > > For example: > >>> from datetime import datetime, timedelta > >>> from zoneinfo import ZoneInfo > > >>> NYC = ZoneInfo("America/New_York") > >>> dt0 = datetime(2020, 1, 1, tzinfo=NYC) > >>> dt1 = datetime(2020, 7, 1, tzinfo=NYC) > > >>> print(dt0) > 2020-01-01 00:00:00-05:00 > >>> print(dt1) > 2020-07-01 00:00:00-04:00 > > >>> print(dt0 + timedelta(days=183)) > 2020-07-02 00:00:00-04:00 > >>> print(dt1 + timedelta(days=1)) > 2020-07-02 00:00:00-04:00 > > Note that the two endpoints are identical, despite the fact that > one of them spans a DST transition and the other one doesn't. > Since the input to `normalize` is just a datetime and it's assumed > that this path-dependence would show up as an inconsistency in the > offset, there's nothing we can do here other than to actually have > all the same problems as pytz. > > Of course, there is another option, which is to, rather than > adopting a wrapper around zoneinfo, adopt a wrapper around pytz > that does /not/ follow PEP 495, but instead just deprecates > `pytz`'s API and tells people to turn on the "use zoneinfo" > feature flag. It has the upside of being fully > backwards-compatible, but the downside of prolonging dependence on > pytz. > > Another option is to modify the shims so that `normalize` always > raises an exception instead of a warning (or maybe it raises an > exception for anything except UTC and fixed offsets). In that > case, version 4.0 will /mostly/ just work and start raising > deprecation warnings, but there will be a hard breakage for anyone > who would be negatively affected by the change in semantics. This > /would/ still leave a possible problem in the other direction, though: > > >>> from datetime import datetime, timedelta > >>> from zoneinfo import ZoneInfo > >>> import pytz > >>> NYC_p = pytz.timezone("America/New_York") > >>> NYC = ZoneInfo("America/New_York") > > >>> dtp_0 = NYC_p.localize(datetime(2020, 10, 31, 12)) > >>> dtp_1 = NYC_p.localize(datetime(2020, 11, 1, 12)) > >>> (dtp_1 - dtp_0 ) / timedelta(hours=1) > 25.0 > > >>> dtz_0 = datetime(2020, 10, 31, 12, tzinfo=NYC) > >>> dtz_1 = datetime(2020, 11, 1, 12, tzinfo=NYC) > >>> (dtz_1 - dtz_0) / timedelta(hours=1) > 24.0 > > This occurs because localized pytz zones are different tzinfo > objects, and as such comparisons and subtraction use inter-zone > semantics. Of course, you'll have this same problem even with a > "hard break", since unlike invocation of `normalize` and > `localize`, subtraction operations will succeed if you swap out > the attached tzinfo for a zoneinfo tzinfo. > > If we go with any variation of using shim-around-zoneinfo like > pytz-deprecation-shim, I'd say those shims need to be introduced > as a breaking change in Django 4.0. If we go with > shim-around-pytz, I think that can safely be introduced in 3.2 > (though that would /require/ simultaneously adding support for > using zoneinfo, and even then it might /mostly/ force people to > either do the migration in a single huge step or to involve some > wrapper functions for handling the period of time where the time > zone type is not consistent throughout the application). > > Best, > Paul > > On 10/9/20 9:31 AM, Kevin Henry wrote: >> I think that the simplest approach—the one that would result in >> the least amount of total work for both Django and its >> users—would be to adopt Nick's suggestion and just switch to >> zoneinfo in 4.0. The problem is that it's very hard to square >> that with Django's stability policy: "We’ll only break backwards >> compatibility of these APIs without a deprecation process if a >> bug or security hole makes it completely unavoidable." >> >> If we're going to follow the deprecation process, then there >> needs to be some overlap where both ways of doing things are >> possible. The shims package is a promising approach, but the fact >> that it's not actually backwards compatible with pytz is a >> serious problem. Adopting it directly as Carlton proposes also >> seems to violate the stability policy, albeit in a less severe way. >> >> >> Kevin >> >> On Thursday, October 8, 2020 at 11:35:21 PM UTC-7 >> smi...@gmail.com wrote: >> >> Hi All, >> >> While I understand the desire to have an early opt-in for >> some I think the important question here is the deprecation >> warnings. The recent URL() change showed that no matter how >> long there is a new way some?/many? folk won't change until >> they need to. >> >> Nick -- if we introduced a breaking change in 4.0, would that >> not have the same impact on folk upgrading to 4.2LTS from >> 3.2LTS as that which Carlton is concerned about (3.2 from >> 2.2), albeit a few years further into the future. >> >> >> David >> >> On Thursday, 8 October 2020 at 09:08:50 UTC+1 >> jure.er...@gmail.com wrote: >> >> I would definitely be in favor of an opt-in: it would >> give developers time to move to the new system at their >> convenience. >> >> Example: we're about to try and tackle the TZ issue in >> our apps and we want to do it "globally" with one >> definitive solution. I'd much rather do it on a library >> that is currently favoured, but not yet default than on a >> deprecated one, even if it's not yet officially >> deprecated. We do have some "import pytz", but currently >> they are few. Once we have a proper approach to handling >> timezone stuff, there's likely going to be more of >> them... or less, depending on the solution ;-) >> >> LP, >> Jure >> >> On 7. 10. 20 17:25, Paul Ganssle wrote: >>> >>> This sounds like a reasonable timeline to me. I think >>> the breakage will be relatively small because I suspect >>> many end-users don't really even know to use `normalize` >>> in the first place, and when introducing the shim into a >>> fundamental library at work I did not get a huge number >>> of breakages, but I am still convinced that it is >>> reasonably categorized as a breaking change. >>> >>> I do think that there's one additional stage that we >>> need to add here (and we chatted about this on twitter a >>> bit), which is a stage that is fully backwards >>> compatible where Django supports using non-pytz zones >>> for users who bring their own time zone. I suspect that >>> will help ease any breaking pain between 3.2 and 4.0, >>> because no one would be forced to make any changes, but >>> end users could proactively migrate to zoneinfo for a >>> smoother transition. >>> >>> I think most of what needs to be done is already in my >>> original PR, it just needs a little conditional logic to >>> handle pytz as well as the shim. >>> >>> I am not sure how you feel about feature flags, but as a >>> "nice to have", I imagine it would also be possible to >>> add a feature flag that opts you in to `zoneinfo` as >>> time zone provider even in 3.2, so that people can jump >>> straight to the 5.0 behavior if they are ready for it. >>> >>> I should be able to devote some time to at least the >>> first part — making Django compatible with zoneinfo even >>> if not actively using it — but likely not for a few >>> weeks at minimum. If anyone wants to jump on either of >>> these ahead of me I don't mind at all and feel free to >>> ping me for review. >>> >>> Best, >>> Paul >>> >>> On 10/7/20 10:48 AM, Carlton Gibson wrote: >>>> Hi Paul. >>>> >>>> Thanks for the input here, and for your patience >>>> >>>> > I am fairly certain this is going to be a tricky >>>> migration and will inevitably come with /some/ user >>>> pain. I don't think this will be Python 2 → 3 style >>>> pain, but some users who have been doing the "right >>>> thing" with pytz will need to make changes to their >>>> code in the long run, which is unfortunate. >>>> >>>> Looking at all the docs, your migration guide on >>>> pytz_deprecation_shim, the example Kevin gave >>>> <https://repl.it/@severian/pytzshim#main.py>, where we >>>> do some arithmetic in a local timezone, and call >>>> `normalize()` in case we crossed a DST boundary, >>>> there's no way we can do this without forcing a >>>> breaking change somewhere. >>>> >>>> So, probably, I've been staring at this too long today, >>>> but I think we should introduce the shim in Django 4.0. >>>> Django 3.2, the next major release will be an LTS. If >>>> we hold-off introducing the change until 4.0, we can >>>> flag it as a breaking change in the 4.0 release notes, >>>> with big warnings, allowing folks extra time to hang >>>> out on the previous LTS if they need it. >>>> >>>> What I wouldn't want to do is to bring the breaking >>>> change in in Django 3.2, because we'll have a whole >>>> load of folks updating from the 2.2 LTS at about the >>>> time when it goes End of Life, and with no warning, >>>> that'd be a hard breaking change to throw on top of >>>> their other issues. >>>> >>>> We'd keep the shim in place for the entire 4.x series, >>>> removing in Django 5.0 as per the deprecation policy >>>> >>>> <https://docs.djangoproject.com/en/3.1/internals/release-process/#deprecation-policy>. >>>> >>>> I think the advantages of doing it this way are two-fold: >>>> >>>> * We allow people to focus on the semantic breaking >>>> change (in folds) separately from the code changes per >>>> se — the logic may have changed slightly in these >>>> cases, but it'll still run. >>>> * It looks easier to migrate Django's code vs branching >>>> on a new setting. (I didn't think through exactly what >>>> that might look like, so happy to see a PoC from anyone.) >>>> >>>> I'm more attached to the timeline (i.e. making the >>>> change after the next LTS) than whether we use the >>>> deprecation shim or not, but can I ask others to give >>>> this their thought too? >>>> >>>> Thanks again! >>>> >>>> Kind Regards, >>>> >>>> Carlton >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to >>>> the Google Groups "Django developers (Contributions to >>>> Django itself)" group. >>>> To unsubscribe from this group and stop receiving >>>> emails from it, send an email to >>>> django-develop...@googlegroups.com. >>>> To view this discussion on the web visit >>>> >>>> https://groups.google.com/d/msgid/django-developers/ce04a6b7-4409-4b20-ba30-4cd64dc0cabfn%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/django-developers/ce04a6b7-4409-4b20-ba30-4cd64dc0cabfn%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> -- >>> You received this message because you are subscribed to >>> the Google Groups "Django developers (Contributions to >>> Django itself)" group. >>> To unsubscribe from this group and stop receiving emails >>> from it, send an email to >>> django-develop...@googlegroups.com. >>> To view this discussion on the web visit >>> >>> https://groups.google.com/d/msgid/django-developers/e13e8ae2-5d43-e550-48a4-cb7ad6e699f6%40ganssle.io >>> >>> <https://groups.google.com/d/msgid/django-developers/e13e8ae2-5d43-e550-48a4-cb7ad6e699f6%40ganssle.io?utm_medium=email&utm_source=footer>. >> >> -- >> You received this message because you are subscribed to the >> Google Groups "Django developers (Contributions to Django >> itself)" group. >> To unsubscribe from this group and stop receiving emails from it, >> send an email to django-develop...@googlegroups.com. >> To view this discussion on the web visit >> >> https://groups.google.com/d/msgid/django-developers/b18754a4-c308-492d-b547-6b3c7cdc1442n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/django-developers/b18754a4-c308-492d-b547-6b3c7cdc1442n%40googlegroups.com?utm_medium=email&utm_source=footer>. > > -- > You received this message because you are subscribed to the Google > Groups "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to django-developers+unsubscr...@googlegroups.com > <mailto:django-developers+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/043100f2-fd50-458a-9b31-c52128a534cbn%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/043100f2-fd50-458a-9b31-c52128a534cbn%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4b473c2b-f4e1-2d25-4f35-4695815e4c25%40ganssle.io.
signature.asc
Description: OpenPGP digital signature
