So, for example, you could wrap pytz and keep a shadow copy of the
pytz tzobject inside your tzobject, and use that to determine the
correct behavior whenever a pytz-specific call is made. So when
localize() is called you call localize on your internal object, and
store that pytz EST tzobject. Then when normalize() is called you use
that to get pytz's version of the EDT time.
> This occurs because localized pytz zones are different tzinfo
objects, and as such comparisons and subtraction use inter-zone
semantics.
Thank you for that example, I hadn't considered that. Unfortunately
that is another fundamental incompatibility between pytz and any
shim-around-zoneinfo (here
<https://repl.it/@severian/pytzshim-subtract> is a runnable version).
I can't think of any way around that one.
> Of course, there is another option, which is to, rather than
adopting a wrapper around zoneinfo, adopt a wrapper around pytz that
does not follow PEP 495, but instead just deprecates `pytz`'s API and
tells people to turn on the "use zoneinfo" feature flag.
Agreed, that is the other main option. It has a few advantages:
- It's backwards-compatible.
- Because it's backwards compatible it could be adopted in 3.2,
allowing a complete transition to zoneinfo by 4.2, a full two years
earlier than the shims approach.
- It only requires users to think about the change once, when they
opt in to the new approach. Using the shim means you have to think
about this issue at least twice: once when the shim is dropped in and
you have to figure out if you're affected by the backwards
incompatibilities; and once (or more) when you actually make the
change (or a series of changes) over to the native zoneinfo style.
The main disadvantage—and a real one—is that it's more work for Django.
Cheers,
Kevin
On Friday, October 9, 2020 at 8:06:49 AM UTC-7 Paul Ganssle wrote:
Before looking at alternatives, I wonder if we can just change
the shims package to make it fully backwards compatible? Right
now the shims version of normalize()
<https://github.com/pganssle/pytz-deprecation-shim/blob/47bd4bdd9346cafa6c6d66817082ccce099890ad/src/pytz_deprecation_shim/_impl.py#L265>
is essentially a noop. Paul, couldn't it actually attempt to
adjust the time the way pytz does? Perhaps by wrapping pytz
itself, and calling its normalize() from the corresponding pytz
timezone; or by simply replicating its time-changing logic?
Apologies if that's a naive question.
It is not really possible to make the shims work the same way
because there's not enough information available to determine
whether an adjustment needs to be made. The reason that
`normalize` works is that pytz attaches different `tzinfo`
objects representing fixed offsets (with a reference to the time
zone they represent) to the datetime. If arithmetic creates an
invalid datetime (i.e. a datetime in mid-June 2020 with EST
attached), `normalize` corrects this by attaching a `tzinfo`
representing the correct offset — and it does that by assuming
that the UTC datetime represented by the erroneous fixed offset
is correct. With PEP 495-style zones, you never create those
datetimes with erroneous offsets, so there's no way to tell
whether a correction is required.
For example:
>>> from datetime import datetime, timedelta
>>> from zoneinfo import ZoneInfo
>>> NYC = ZoneInfo("America/New_York")
>>> dt0 = datetime(2020, 1, 1, tzinfo=NYC)
>>> dt1 = datetime(2020, 7, 1, tzinfo=NYC)
>>> print(dt0)
2020-01-01 00:00:00-05:00
>>> print(dt1)
2020-07-01 00:00:00-04:00
>>> print(dt0 + timedelta(days=183))
2020-07-02 00:00:00-04:00
>>> print(dt1 + timedelta(days=1))
2020-07-02 00:00:00-04:00
Note that the two endpoints are identical, despite the fact that
one of them spans a DST transition and the other one doesn't.
Since the input to `normalize` is just a datetime and it's
assumed that this path-dependence would show up as an
inconsistency in the offset, there's nothing we can do here other
than to actually have all the same problems as pytz.
Of course, there is another option, which is to, rather than
adopting a wrapper around zoneinfo, adopt a wrapper around pytz
that does /not/ follow PEP 495, but instead just deprecates
`pytz`'s API and tells people to turn on the "use zoneinfo"
feature flag. It has the upside of being fully
backwards-compatible, but the downside of prolonging dependence
on pytz.
Another option is to modify the shims so that `normalize` always
raises an exception instead of a warning (or maybe it raises an
exception for anything except UTC and fixed offsets). In that
case, version 4.0 will /mostly/ just work and start raising
deprecation warnings, but there will be a hard breakage for
anyone who would be negatively affected by the change in
semantics. This /would/ still leave a possible problem in the
other direction, though:
>>> from datetime import datetime, timedelta
>>> from zoneinfo import ZoneInfo
>>> import pytz
>>> NYC_p = pytz.timezone("America/New_York")
>>> NYC = ZoneInfo("America/New_York")
>>> dtp_0 = NYC_p.localize(datetime(2020, 10, 31, 12))
>>> dtp_1 = NYC_p.localize(datetime(2020, 11, 1, 12))
>>> (dtp_1 - dtp_0 ) / timedelta(hours=1)
25.0
>>> dtz_0 = datetime(2020, 10, 31, 12, tzinfo=NYC)
>>> dtz_1 = datetime(2020, 11, 1, 12, tzinfo=NYC)
>>> (dtz_1 - dtz_0) / timedelta(hours=1)
24.0
This occurs because localized pytz zones are different tzinfo
objects, and as such comparisons and subtraction use inter-zone
semantics. Of course, you'll have this same problem even with a
"hard break", since unlike invocation of `normalize` and
`localize`, subtraction operations will succeed if you swap out
the attached tzinfo for a zoneinfo tzinfo.
If we go with any variation of using shim-around-zoneinfo like
pytz-deprecation-shim, I'd say those shims need to be introduced
as a breaking change in Django 4.0. If we go with
shim-around-pytz, I think that can safely be introduced in 3.2
(though that would /require/ simultaneously adding support for
using zoneinfo, and even then it might /mostly/ force people to
either do the migration in a single huge step or to involve some
wrapper functions for handling the period of time where the time
zone type is not consistent throughout the application).
Best,
Paul
On 10/9/20 9:31 AM, Kevin Henry wrote:
I think that the simplest approach—the one that would result in
the least amount of total work for both Django and its
users—would be to adopt Nick's suggestion and just switch to
zoneinfo in 4.0. The problem is that it's very hard to square
that with Django's stability policy: "We’ll only break backwards
compatibility of these APIs without a deprecation process if a
bug or security hole makes it completely unavoidable."
If we're going to follow the deprecation process, then there
needs to be some overlap where both ways of doing things are
possible. The shims package is a promising approach, but the
fact that it's not actually backwards compatible with pytz is a
serious problem. Adopting it directly as Carlton proposes also
seems to violate the stability policy, albeit in a less severe way.
Kevin
On Thursday, October 8, 2020 at 11:35:21 PM UTC-7
smi...@gmail.com wrote:
Hi All,
While I understand the desire to have an early opt-in for
some I think the important question here is the deprecation
warnings. The recent URL() change showed that no matter how
long there is a new way some?/many? folk won't change until
they need to.
Nick -- if we introduced a breaking change in 4.0, would
that not have the same impact on folk upgrading to 4.2LTS
from 3.2LTS as that which Carlton is concerned about (3.2
from 2.2), albeit a few years further into the future.
David
On Thursday, 8 October 2020 at 09:08:50 UTC+1
jure.er...@gmail.com wrote:
I would definitely be in favor of an opt-in: it would
give developers time to move to the new system at their
convenience.
Example: we're about to try and tackle the TZ issue in
our apps and we want to do it "globally" with one
definitive solution. I'd much rather do it on a library
that is currently favoured, but not yet default than on
a deprecated one, even if it's not yet officially
deprecated. We do have some "import pytz", but currently
they are few. Once we have a proper approach to handling
timezone stuff, there's likely going to be more of
them... or less, depending on the solution ;-)
LP,
Jure
On 7. 10. 20 17:25, Paul Ganssle wrote:
This sounds like a reasonable timeline to me. I think
the breakage will be relatively small because I suspect
many end-users don't really even know to use
`normalize` in the first place, and when introducing
the shim into a fundamental library at work I did not
get a huge number of breakages, but I am still
convinced that it is reasonably categorized as a
breaking change.
I do think that there's one additional stage that we
need to add here (and we chatted about this on twitter
a bit), which is a stage that is fully backwards
compatible where Django supports using non-pytz zones
for users who bring their own time zone. I suspect that
will help ease any breaking pain between 3.2 and 4.0,
because no one would be forced to make any changes, but
end users could proactively migrate to zoneinfo for a
smoother transition.
I think most of what needs to be done is already in my
original PR, it just needs a little conditional logic
to handle pytz as well as the shim.
I am not sure how you feel about feature flags, but as
a "nice to have", I imagine it would also be possible
to add a feature flag that opts you in to `zoneinfo` as
time zone provider even in 3.2, so that people can jump
straight to the 5.0 behavior if they are ready for it.
I should be able to devote some time to at least the
first part — making Django compatible with zoneinfo
even if not actively using it — but likely not for a
few weeks at minimum. If anyone wants to jump on either
of these ahead of me I don't mind at all and feel free
to ping me for review.
Best,
Paul
On 10/7/20 10:48 AM, Carlton Gibson wrote:
Hi Paul.
Thanks for the input here, and for your patience
> I am fairly certain this is going to be a tricky
migration and will inevitably come with /some/ user
pain. I don't think this will be Python 2 → 3 style
pain, but some users who have been doing the "right
thing" with pytz will need to make changes to their
code in the long run, which is unfortunate.
Looking at all the docs, your migration guide on
pytz_deprecation_shim, the example Kevin gave
<https://repl.it/@severian/pytzshim#main.py>, where we
do some arithmetic in a local timezone, and call
`normalize()` in case we crossed a DST boundary,
there's no way we can do this without forcing a
breaking change somewhere.
So, probably, I've been staring at this too long
today, but I think we should introduce the shim in
Django 4.0. Django 3.2, the next major release will be
an LTS. If we hold-off introducing the change until
4.0, we can flag it as a breaking change in the 4.0
release notes, with big warnings, allowing folks extra
time to hang out on the previous LTS if they need it.
What I wouldn't want to do is to bring the breaking
change in in Django 3.2, because we'll have a whole
load of folks updating from the 2.2 LTS at about the
time when it goes End of Life, and with no warning,
that'd be a hard breaking change to throw on top of
their other issues.
We'd keep the shim in place for the entire 4.x series,
removing in Django 5.0 as per the deprecation policy
<https://docs.djangoproject.com/en/3.1/internals/release-process/#deprecation-policy>.
I think the advantages of doing it this way are two-fold:
* We allow people to focus on the semantic breaking
change (in folds) separately from the code changes per
se — the logic may have changed slightly in these
cases, but it'll still run.
* It looks easier to migrate Django's code vs
branching on a new setting. (I didn't think through
exactly what that might look like, so happy to see a
PoC from anyone.)
I'm more attached to the timeline (i.e. making the
change after the next LTS) than whether we use the
deprecation shim or not, but can I ask others to give
this their thought too?
Thanks again!
Kind Regards,
Carlton
--
You received this message because you are subscribed
to the Google Groups "Django developers (Contributions
to Django itself)" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to
django-develop...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/ce04a6b7-4409-4b20-ba30-4cd64dc0cabfn%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/ce04a6b7-4409-4b20-ba30-4cd64dc0cabfn%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to
the Google Groups "Django developers (Contributions to
Django itself)" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to
django-develop...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/e13e8ae2-5d43-e550-48a4-cb7ad6e699f6%40ganssle.io
<https://groups.google.com/d/msgid/django-developers/e13e8ae2-5d43-e550-48a4-cb7ad6e699f6%40ganssle.io?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the
Google Groups "Django developers (Contributions to Django
itself)" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/b18754a4-c308-492d-b547-6b3c7cdc1442n%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/b18754a4-c308-492d-b547-6b3c7cdc1442n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to django-developers+unsubscr...@googlegroups.com
<mailto:django-developers+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/043100f2-fd50-458a-9b31-c52128a534cbn%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/043100f2-fd50-458a-9b31-c52128a534cbn%40googlegroups.com?utm_medium=email&utm_source=footer>.
