Re: [Python-Dev] PEP 447 (type.__getdescriptor__)

[Ronald Oussoren]

I’ve pushed a minor update to the PEP to the repository. The benchmark results 
are still out of date, I need want to run those on an idle machine to get 
reliable results.

The PEP has one significant change w.r.t. the previous version: it now requires 
the use of a new type flag to enable the usage of the new slot in C code. This 
is due to concerns that loading old extensions might crash the interpreter 
otherwise.

References to earlier discussions (also added to the PEP):

* http://marc.info/?l=python-dev&m=137510220928964&w=2
* https://mail.python.org/pipermail/python-ideas/2014-July/028420.html
* https://mail.python.org/pipermail/python-dev/2013-July/127321.html

And finally, I’ve updated the implementation in issue 18181. The implementation 
passes the test suite with the current trunk and is good enough to play around 
with.  There is still an important issue though: I’ve done some micro 
benchmarking and those indicate that the method_cache mechanism in typeobject.c 
doesn’t work with my changes and that has a clear performance impact and must 
be fixed. That shouldn’t be too hard to fix, it’s probably just a botched check 
before the blocks of code that use and update the cache.

Ronald

> On 24 Jul 2015, at 19:55, Ronald Oussoren  wrote:
> 
>> 
>> On 24 Jul 2015, at 17:29, Nick Coghlan > > wrote:
>> 
>> On 25 July 2015 at 00:50, Brett Cannon > > wrote:
>>> Leave the decorator out like __new__, otherwise people are bound to forget
>>> it and have a hard time debugging why their code doesn't work.
>> 
>> I'd actually advocate for keeping this as a metaclass method, rather
>> than making it available to any type instance. The key thing to
>> consider for me is "What additional power does making it a method on
>> the class itself grant to mixin types?”
> 
> To be honest, I hadn’t considered mixin types yet. 
> 
>> 
>> With PEP 487, the __init_subclass__ proposal only grants mixins the
>> power to implicitly run additional code when new subclasses are
>> defined. They have no additional ability to influence the behaviour of
>> the specific class adding the mixin into the inheritance hierarchy.
>> 
>> With PEP 447, as currently written, a mixin that wants to alter how
>> descriptors are looked up will be able to do so implicitly as long as
>> there are no other custom metaclasses in the picture. As soon as there
>> are *two* custom metaclasses involved, you'll get an error at
>> definition time and have to sort out how you want the metaclass
>> inheritance to work and have a chance to notice if there are two
>> competing __getdescriptor__ implementations.
>> 
>> However, if __getdescriptor__ moves to being a class method on object
>> rather than an instance method on type, then you'll lose that
>> assistance from the metaclass checker - if you have two classes in
>> your MRO with mutually incompatible __getdescriptor__ implementations,
>> you're likely to be in for a world of pain as you try to figure out
>> the source of any related bugs.
> 
> That’s a good point, and something that will move something that I’ve 
> wanted to look into forward on my list: the difference between a
> classmethod and a method on the class defined through a metaclass.
> 
> The semantics I’d like to have is that __getdescriptor__ is a local decision,
> defining __getdescriptor__ for a class should only affect that class and its
> subclass, and shouldn’t affect how superclasses are handled by 
> __getattribute__.
> That is something that can be done by defining __getdescriptor__ on a 
> metaclass,
> and AFAIK requires active cooperation when using a @classmethod.
> 
> It should be possible to demonstrate the differences in a pure Python
> prototype. 
> 
> Ronald
> 
>> 
>> Cheers,
>> Nick.
>> 
>> -- 
>> Nick Coghlan   |   ncogh...@gmail.com    |   
>> Brisbane, Australia
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org 
>> https://mail.python.org/mailman/listinfo/python-dev 
>> 
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com 
>> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[Alexander Belopolsky]

On Sat, Jul 25, 2015 at 2:40 AM, Lennart Regebro  wrote:

> There really is a reason every other date time implementation I know
> of uses UTC internally, and there really is a reason why everyone
> always recommends storing date times in UTC with the time zone or
> offset separately.
>

Current datetime design does not prevent your application from storing
date-times
in UTC.  You can store them in naive datetime instances, but the
recommended
approach is to use datetime instances with tzinfo=timezone.utc.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 447 (type.__getdescriptor__)

[Mark Shannon]

Hi,

On 22/07/15 09:25, Ronald Oussoren wrote:> Hi,
> 
> Another summer with another EuroPython, which means its time again to 
> try to revive PEP 447…
> 

IMO, there are two main issues with the PEP and implementation.

1. The implementation as outlined in the PEP is infinitely recursive, since the
lookup of "__getdescriptor__" on type must necessarily call
type.__getdescriptor__.
The implementation (in C) special cases classes that inherit "__getdescriptor__"
from type. This special casing should be mentioned in the PEP.

2. The actual implementation in C does not account for the case where the class
of a metaclass implements __getdescriptor__ and that method returns a value when
called with "__getdescriptor__" as the argument.



Why was "__getattribute_super__" rejected as an alternative? No reason is given.

"__getattribute_super__" has none of the problems listed above.
Making super(t, obj) delegate to t.__super__(obj) seems consistent with other
builtin method/classes and doesn't add corner cases to the already complex
implementation of PyType_Lookup().

Cheers,
Mark
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[Ryan Hiebert]

> On Jul 25, 2015, at 09:15, Alexander Belopolsky 
>  wrote:
> 
> 
>> On Sat, Jul 25, 2015 at 2:40 AM, Lennart Regebro  wrote:
>> There really is a reason every other date time implementation I know
>> of uses UTC internally, and there really is a reason why everyone
>> always recommends storing date times in UTC with the time zone or
>> offset separately.
> 
> Current datetime design does not prevent your application from storing 
> date-times
> in UTC.  You can store them in naive datetime instances, but the recommended 
> approach is to use datetime instances with tzinfo=timezone.utc.

Yes, and now he wants to do the same thing for the internals of the datetime 
module, for the same reasons that it's the best thing most everywhere else. 
It's just going to take some significant effort to make that happen.___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Burning down the backlog.

[Robert Collins]

On 21 July 2015 at 19:40, Nick Coghlan  wrote:

> All of this is why the chart that I believe should be worrying people
> is the topmost one on this page:
> http://bugs.python.org/issue?@template=stats
>
> Both the number of open issues and the number of open issues with
> patches are steadily trending upwards. That means the bottleneck in
> the current process *isn't* getting patches written in the first
> place, it's getting them up to the appropriate standards and applied.
> Yet the answer to the problem isn't a simple "recruit more core
> developers", as the existing core developers are *also* the bottleneck
> in the review and mentoring process for *new* core developers.

Those charts doesn't show patches in 'commit-review' -
http://bugs.python.org/issue?%40columns=title&%40columns=id&stage=5&%40columns=activity&%40sort=activity&status=1&%40columns=status&%40pagesize=50&%40startwith=0&%40sortdir=on&%40action=search

There are only 45 of those patches.

AIUI - and I'm very new to core here - anyone in triagers can get
patches up to commit-review status.

I think we should set a goal to keep inventory low here - e.g. review
and either bounce back to patch review, or commit, in less than a
month. Now - a month isn't super low, but we have lots of stuff
greater than a month.

For my part, I'm going to pick up more or less one thing a day and
review it, but I think it would be great if other committers were to
also to do this: if we had 5 of us doing 1 a day, I think we'd burn
down this 45 patch backlog rapidly without significant individual
cost. At which point, we can fairly say to folk doing triage that
we're ready for patches :)

-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[ISAAC J SCHWABACHER]

> From: Tim Peters 
> Sent: Friday, July 24, 2015 20:39
> To: ISAAC J SCHWABACHER
> Cc: Alexander Belopolsky; Lennart Regebro; Python-Dev
> Subject: Re: [Python-Dev] Status on PEP-431 Timezones
>
> [ISAAC J SCHWABACHER ]
> > ...
> > I disagree with the view Tim had of time zones when he wrote that comment
> > (and that code). It sounds like he views US/Eastern and US/Central as time
> > zones (which they are), but thinks of the various America/Indiana zones as
> > switching back and forth between them, rather than being time zones in their
> > own right
>
> You can think of them anyway you like.  The point of the code was to
> provide a simple & efficient way to convert from UTC to local time in
> all "time zones" in known actual use at the time; the point of the
> comment was to explain the limitations of the code.  Although, as
> Allexander noted, the stated assumptions are stronger than needed.
>
> > I think the right perspective is that a time zone *is* the function that its
> > `fromutc()` method implements,
>
> Fine by me ;-)

My issue is that you're computing `fromutc()`, which is a function, in terms of 
`dst()` and `utcoffset()`, which aren't. I think this is backwards; `dst()` and 
`utcoffset()` should be computed from `fromutc()` plus some additional 
information that has to be present anyway in order to implement `fromutc()`. 
With the extra bit, `dst()` and `utcoffset()` become partial functions, which 
makes it *possible* to get the right answer in all cases, but it's still 
simpler to start with the total function and work from there.

> > although of course we need additional information in order to actually
> > compute (rather than merely mathematically define) its inverse. Daylight 
> > Saving
> > Time is a red herring,
>
> Overstated.  DST is in fact the _only_ real complication in 99.99% of
> time zones (perhaps even 99.9913% ;-) ).  As the docs say, if you have
> some crazy-ass time zone in mind, fine, that's why fromutc() was
> exposed (so your; crazy-ass tzinfo class can override it).

I stand by what I meant by this, even if I did a bad job of expressing the 
point. Assuming that all time zone discontinuities are due to DST changes 
breaks many time zones (really almost all of the Olson time zones, though only 
for a vanishingly small fraction of datetimes), but that's not the point I was 
making. The point is that it doesn't buy us anything. Though this is probably 
obscured by all the markup, the more general algorithm I gave is also simpler 
than the one in the comment in datetime.py, and the reason for that is that it 
solves an easier problem, but one that serves our practical purposes just as 
well.

> > and assumptions 2 and 4
>
> Nitpick:  4 is a consequence of 2, not an independent assumption.
>
> > in that exposition are just wrong from this point of view.
>
> As above, there is no particular POV in this code:  just a specific
> fromutc() implementation, comments that explain its limitations, and
> an invitation in the docs to override it if it's not enough for your
> case.

I went too far in inferring your viewpoint from your code.  I don't find fault 
with the explanation on its own terms. But adding zoneinfo to the stdlib, as 
PEP 431 proposes to do, requires making weaker assumptions and asking a 
different question than the one answered in the comment. 

> > In the worst case, Asia/Riyadh's two years of solar time completely shatter
> > these assumptions.
>
> Sure.  But, honestly, who cares?  Riyadh Solar Time was so
> off-the-wall that even the Saudis gave up on it 25 years ago (after a
> miserable 3-year experiment with it).  "Practicality beats purity".

As a mathematician at heart, I have a deep and abiding conviction, which I 
expect nobody else to share, that purity begets practicality in the long run. 
At least if you've found the right abstraction.

> > [eliding a more-general view of what time zones "really" are]

[note for people just joining this conversation: I think the information in the 
elision is critical to understanding what I'm talking about]

> I'm not eliding it because I disagree with it, but because time zones
> are political constructions.  "The math" we make up may or may not be
> good enough to deal with all future political abominations; for
> example:
>
> > ...
> > This assumption would be violated if, for example, some jurisdiction
> > decided to fall back two hours by falling back one hour and then
> > immediately falling back a second hour.  I recommend the overthrow
> > of any such jurisdiction and its (annexation by the Netherlands)[3].
>
> That's not objectively any more bizarre than Riyadh Solar Time.
> Although, if I've lived longer than you, I may be more wary about the
> creative stupidity of political schemes ;-)

It's not, you have, and you probably are. :)

But these assumptions didn't come out of nowhere. They're the assumptions 
behind zoneinfo, weakened as much as possible without making the problem any 
harder. It's hard t

Re: [Python-Dev] Status on PEP-431 Timezones

[Tim Peters]

[Lennart Regebro ]
>>> And I would want to remind everyone again that this is not a question
>>> of the problem being impossible. It's just really complex to get right
>>> in all cases, and that always having the UTC timestamp around gets rid
>>> of most of that complexity.

[Tim]
>> Could you please be explicit about what "the problem" is?  Everyone
>> here is guessing at what you think "the problem" is.

[Lennart]
> The problem is that it is exceedingly complicated to get all the
> calculations back and forth between local time and UTC to be correct
> at all times and for all cases. It really doesn't get more specific
> than that. I don't remember which exact problem it was that made me
> decide that this was not the correct solution and that we should use
> UTC internally, but I don't think that matters, because I'm also sure
> that it was not the last case, as I was far from near the end in
> adding testcases.

I believe everyone here is saying it "shouldn't be" exceedingly
complicated, or even particularly hard, if you add the is_dst flags
the PEP says it would add.

But is the PEP complete?  Under the "Postponement" section, it says:

The implementation has turned out to be exceedingly complex,
due to having to convert back and forth between the local time
and UTC during arithmetic and adjusting the DST for each
arithmetic step, with ambiguous times being particularly hard
to get right.

However, the _body_ of the PEP said nothing whatsoever about altering
arithmetic.  The body of the PEP sounds like it's mainly just
proposing to fold the pytz package into the core.  Perhaps doing
_just_ that much would get this project unstuck?  Hope springs eternal
:-)

> Once again I'm sure it's not impossible to somehow come up with an
> implementation and an API that can do this based on local time, but
> once again I am of the opinion that it is the wrong thing to do. We
> should switch to using UTC internally, because that will make
> everything so much simpler.

Like what?  I'm still looking for a concrete example of what "the
problem" is (or even "a" problem).

> I am in no way against other people implementing this PEP, but I think
> you will end up with very complex code that will be hard to maintain.

Somebody first needs to define what "the problem" is ;-)

> There really is a reason every other date time implementation I know
> of uses UTC internally,

Yes, but the fundamental reason datetime does not is that Guido
consciously and deliberately decided that "naive datetime" would be
most useful most often for most users.  That's why "naive" objects are
the default.  And even for "aware" objects, arithmetic staying within
a single time zone was deliberately specified to be "naive" too.  My
guess is that all other datetime implementations you know of have no
concept of "naive" datetimes. let alone make naive datetimes primary.
Small wonder, if so, that they're all different in this way.

That's a design decision not everyone likes, and certainly isn't
suitable for all purposes, but the debate over that ended a dozen
years ago when the decision was made.  If your vision of PEP 431
_changes_ that design decision (which it sure _sounds_ like it wants
to based on what you're typing here, but which PEP 431 itself does not
appear to say - impossible to tell which from here without any
specific example(s)), that may account for all sorts of complications
that aren't apparent to me.

> and there really is a reason why everyone always recommends storing date
> times in UTC with the time zone or offset separately.

Well, that's the second thing they recommend - and they can already do
that.  The first thing to recommend is to use naive objects in any
application where that's possible, so that you don't have to bother
with _any_ time zone esoterica, surprises, complications or overheads.
After all, it's 7:54 PM as I type this, and that's perfectly clear to
me ;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[Lennart Regebro]

On Sun, Jul 26, 2015 at 2:56 AM, Tim Peters  wrote:
> However, the _body_ of the PEP said nothing whatsoever about altering
> arithmetic.  The body of the PEP sounds like it's mainly just
> proposing to fold the pytz package into the core.  Perhaps doing
> _just_ that much would get this project unstuck?  Hope springs eternal
> :-)

The pytz package has an API and a usage that is different from the
datetime() module. One of the things you need to do is that after each
time you do arithmetic, you have to normalize the result. This is done
because the original API design did not realize the difficulties and
complexities of timezone handling and therefore left out things like
ambiguous times.

The PEP attemps to improved the datetime modules API so that it can
handle the ambiguous times. It also says that the implementation will
be based on pytz, because it was my assumption that this would be
easy, since pytz already handles ambiguous times. During my attempt of
implementing it I realized it wasn't easy at all, and it wasn't as
easy as folding pytz into the core.

Yes, the PEP gives that impression, because that was the assumption
when I wrote the draft. Just folding pytz into the core without
modifying the API defeats the whole purpose of the PEP, since
installing pytz is a trivial task.

> Like what?  I'm still looking for a concrete example of what "the
> problem" is (or even "a" problem).

A problem is that you have a datetime, and add a timedelata to it, and
it should then result in a datetime that is actually that timedelta
later. And if you subtract the same timedelta from the result, it
should return a datetime that is equal to the original datetime.

This sounds ridiculously simple, and is ridiculously difficult to make
happen in all cases that we want to support (Riyahd time zone and leap
seconds not included). That IS the specific, concrete problem, and if
you don't believe me, there is nothing I can do to convince you.
Perhaps I am a complete moron and simply incompetent to do this, and
in that case I'm sure you could implement this over a day, and then
please do so, but for the love of the founders of computing I'm not
going to spend more time repeating it on this mailing list, because
then we would do better in having you implement this instead of
reading emails. Me repeating this a waste of time for everyone
involved, and I will now stop.

> 

I was not involved in the discussion then, and even if I had been,
that's still before I knew anything about the topic. I don't know what
the arguments were, and I don't think it's constructive to try to
figure out exactly why that decision was made. That is all to similar
to assigning blame, which only makes people feel bad. Those who get
blamed feel bad, and those who blame feel like dicks and onlookers get
annoyed. Let us look forward instead.

I am operating both without any need to defend that decision, as I was
not involved in it, and I am operating with 20/20 hindsight as I am
one of the few people having tried to implement a timezone
implementation that supports ambiguous datetimes based on that
decision. And then it is perfectly clear and obvious that the decision
was a mistake and that we should rectify it.

The only question for me is how and when.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[David Mertz]

At the risk of being off-topic, I realize I really DO NOT currently
understand datetime in its current incarnation.  It's too bad PEP 431
proves so difficult to implement.

Even using `pytz` is there any way currently to get sensible answers to,
e.g.:

from datetime import *
from pytz import timezone
pacific = timezone('US/Pacific')
pacific.localize(datetime(2015, 11, 1, 1, 30))  # Ambiguous time
pacific.localize(datetime(2015, 3, 8, 2, 30))   # Non-existent time


That is, what if I had *not* just looked up when the time change happens,
and was innocently trying to define one of those datetimes above?  Is there
ANY existing way to have an error raised—or check in some other way—for the
fact that one of the times occurs twice on my clock, and the other never
occurs at all?


On Sat, Jul 25, 2015 at 8:31 PM, Lennart Regebro  wrote:

> On Sun, Jul 26, 2015 at 2:56 AM, Tim Peters  wrote:
> > However, the _body_ of the PEP said nothing whatsoever about altering
> > arithmetic.  The body of the PEP sounds like it's mainly just
> > proposing to fold the pytz package into the core.  Perhaps doing
> > _just_ that much would get this project unstuck?  Hope springs eternal
> > :-)
>
> The pytz package has an API and a usage that is different from the
> datetime() module. One of the things you need to do is that after each
> time you do arithmetic, you have to normalize the result. This is done
> because the original API design did not realize the difficulties and
> complexities of timezone handling and therefore left out things like
> ambiguous times.
>
> The PEP attemps to improved the datetime modules API so that it can
> handle the ambiguous times. It also says that the implementation will
> be based on pytz, because it was my assumption that this would be
> easy, since pytz already handles ambiguous times. During my attempt of
> implementing it I realized it wasn't easy at all, and it wasn't as
> easy as folding pytz into the core.
>
> Yes, the PEP gives that impression, because that was the assumption
> when I wrote the draft. Just folding pytz into the core without
> modifying the API defeats the whole purpose of the PEP, since
> installing pytz is a trivial task.
>
> > Like what?  I'm still looking for a concrete example of what "the
> > problem" is (or even "a" problem).
>
> A problem is that you have a datetime, and add a timedelata to it, and
> it should then result in a datetime that is actually that timedelta
> later. And if you subtract the same timedelta from the result, it
> should return a datetime that is equal to the original datetime.
>
> This sounds ridiculously simple, and is ridiculously difficult to make
> happen in all cases that we want to support (Riyahd time zone and leap
> seconds not included). That IS the specific, concrete problem, and if
> you don't believe me, there is nothing I can do to convince you.
> Perhaps I am a complete moron and simply incompetent to do this, and
> in that case I'm sure you could implement this over a day, and then
> please do so, but for the love of the founders of computing I'm not
> going to spend more time repeating it on this mailing list, because
> then we would do better in having you implement this instead of
> reading emails. Me repeating this a waste of time for everyone
> involved, and I will now stop.
>
> > 
>
> I was not involved in the discussion then, and even if I had been,
> that's still before I knew anything about the topic. I don't know what
> the arguments were, and I don't think it's constructive to try to
> figure out exactly why that decision was made. That is all to similar
> to assigning blame, which only makes people feel bad. Those who get
> blamed feel bad, and those who blame feel like dicks and onlookers get
> annoyed. Let us look forward instead.
>
> I am operating both without any need to defend that decision, as I was
> not involved in it, and I am operating with 20/20 hindsight as I am
> one of the few people having tried to implement a timezone
> implementation that supports ambiguous datetimes based on that
> decision. And then it is perfectly clear and obvious that the decision
> was a mistake and that we should rectify it.
>
> The only question for me is how and when.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40m

Re: [Python-Dev] Status on PEP-431 Timezones

[Tim Peters]

[ISAAC J SCHWABACHER ]
>>> ...
>>> I think the right perspective is that a time zone *is* the function that its
>>> `fromutc()` method implements,

[Tim]
>> Fine by me ;-)

[Isaac]
> My issue is that you're computing `fromutc()`, which is a function, in
> terms of `dst()` and `utcoffset()`, which aren't.

I know.  That's not "an issue" that will gain traction, though ;-)

> I think this is backwards; `dst()` and `utcoffset()` should be computed
> from `fromutc()` plus some additional information that has to be present
> anyway in order to implement `fromutc()`.

Memory lane:  that additional information doesn't exist now.  I think
it "should have", but at the time, as I recall there was fatal
opposition to storing an `isdst` flag because it would consume an
extra byte in the pickle format.  That was enough to kill it:
datetime development was paid for by a company very concerned about
pickle sizes ;-)

> With the extra bit, `dst()` and `utcoffset()` become partial functions, which
> makes it *possible* to get the right answer in all cases, but it's still 
> simpler
> to start with the total function and work from there.

Well, maybe simpler for you, but I think not in general.  At the time,
all aspects of datetime's development were vigorously debated, but
mostly on Zope Corp (the company paying for it) wikis and mailing
lists.  While some people didn't care about time zones at all, most
did.  Of the latter:

- All were keenly aware of the need to incorporate UTC offsets.
- All were keenly aware of the need to accommodate "daylight time" schemes.
- None gave a fig about anything else.

Very late in the game, examples were given of localities that had in
fact changed their UTC offsets from time to time, but as curiosities
rather than as "issues".  That's when I created fromutc() - it was a
last-second addition.  I cared enough to make it _possible_ to
accommodate such cases, but there was no interest (or time) to cater
to them directly.  Instead fromutc() was written to use only the
already-existing utcoffset() and dst().  Everyone already knew how to
use the latter:  they directly corresponded to the two things everyone
cared about keenly from the start.

That doesn't _preclude_ anyone from writing a more-general fromutc(),
and I encourage, for example, you to do so ;-)  I agree it's the most
fundamental thing from an abstract mathematical view, but "UTC offset"
and "DST offset" fit most peoples' brains a hell of a lot better than
"collection of piecewise continuous monotonically increasing functions
whose images don't overlap too much" ;-)

>>>  Daylight Saving Time is a red herring,

>> Overstated 

> I stand by what I meant by this, even if I did a bad job of expressing
> the point. Assuming that all time zone discontinuities are due to DST
> changes breaks many time zones (really almost all of the Olson time
> zones, though only for a vanishingly small fraction of datetimes),

It's use cases that are missing here:  who needs to convert historic
times to/from UTC. and where the "problem times" are generally
arranged by politicians to occur when most people are sleeping?
That's why nobody really cared about offset-changing zones at the
start.  Yes, such zones exist, but times recorded in such zones are in
yesterday's databases we don't care about anymore except maybe to
display the values.

> but that's not the point I was making. The point is that it doesn't buy us
> anything.

Au contraire:  as above, it bought datetime a model people thought
they understood at once, since almost everyone has wrestled with UTC
offsets and daylight-time switches in ordinary daily life.  Implement
utcoffset() and dst(), and you're done.  Even if you're really not,
you _think_ you are, so you slumber peacefully then ;-)

> Though this is probably obscured by all the markup, the more general
> algorithm I gave is also simpler than the one in the comment in datetime.py,
> and the reason for that is that it solves an easier problem, but one that
> serves our practical purposes just as well.

It's heavily obscured by the heavy markup.  Write some Python code
instead?  I expect few people will try to untangle the meaning
otherwise.

As for whether it's simpler - eh, don't know.  Here's the actual code,
stripped of error-checking:

def fromutc(self, dt):
dtoff = dt.utcoffset()
dtdst = dt.dst()
delta = dtoff - dtdst
if delta:
dt += delta
dtdst = dt.dst()
return dt + dtdst

Will your code run faster?  Have fewer conditionals?  Fewer lines?
Shorter lines?  Less nesting?  Fewer operations?  Important to me,
though, is that your code should be far more self-evidently _correct_,
provided the reader understands the math underlying it (which will
require - as this code does - referring to a relatively massive wall
of text to explain it).

> ...
> I went too far in inferring your viewpoint from your code.  I don't find fault
> with the explanation on its own ter

Re: [Python-Dev] Status on PEP-431 Timezones

[Tim Peters]

[Tim]
>> However, the _body_ of the PEP said nothing whatsoever about altering
>> arithmetic.  The body of the PEP sounds like it's mainly just
>> proposing to fold the pytz package into the core.  Perhaps doing
>> _just_ that much would get this project unstuck?  Hope springs eternal :-)

[Lennart Regebro ]
> The pytz package has an API and a usage that is different from the
> datetime() module. One of the things you need to do is that after each
> time you do arithmetic, you have to normalize the result. This is done
> because the original API design did not realize the difficulties and
> complexities of timezone handling and therefore left out things like
> ambiguous times.

Oh, they were realized - indeed, the pytz docs point to Python's
tzinfo docs to explain the ambiguities, and the latter docs existed
before ;-) day 1.

The Python docs also are quite clear about that all arithmetic within
a single timezone is "naive".  That was intentional.  The _intended_
way to do "aware" arithmetic was always to convert to UTC, do the
arithmetic, then convert back.

You never _have_ to normalize() in pytz.  But it's needed if you
_don't_ follow pytz's explicit

The preferred way of dealing with times is to always work in UTC,
converting to localtime only when generating output to be read by
humans

advice, and want to do "aware" arithmetic directly in a non-UTC time
zone.  Python's datetime never intended to support that directly.
Quite the contrary.  I know people who feel otherwise tend to think of
that as a lazy compromise (or some such), but naive arithmetic was
intended to be "a feature".  Fight the design every step of the way,
and, yup, you get problems every step of the way.

> The PEP attemps to improved the datetime modules API so that it can
> handle the ambiguous times.

No problem with that.  I always thought the lack of storing
is_dst-like info was datetime's biggest wart.

> It also says that the implementation will be based on pytz, because it
> was my assumption that this would be easy, since pytz already handles
> ambiguous times. During my attempt of implementing it I realized it
> wasn't easy at all, and it wasn't as easy as folding pytz into the core.

Is it the case that pytz also "fails" in the cases your attempts "fail"?

In any case, if you're trying to change how "aware" datetime
arithmetic works, that's a major and backward-incompatible change.
Does Guido realize it?  As before, it's not at all clear from the PEP.

> Yes, the PEP gives that impression, because that was the assumption
> when I wrote the draft. Just folding pytz into the core without
> modifying the API defeats the whole purpose of the PEP, since
> installing pytz is a trivial task.

"Batteries included" has some attractions all on its own.  On top of
that, adding is_dst-like flags to appropriate methods may have major
attractions.  Changing the semantics of datetime arithmetic has major
attractions to some people, but also major drawbacks - regardless,
since changing it turns Guido's original design on its head, he really
needs to Pronounce on that part.

>> Like what?  I'm still looking for a concrete example of what "the
>> problem" is (or even "a" problem).

> A problem is that you have a datetime, and add a timedelata to it, and
> it should then result in a datetime that is actually that timedelta
> later. And if you subtract the same timedelta from the result, it
> should return a datetime that is equal to the original datetime.
>
> This sounds ridiculously simple

Ah, but it already happens that way - because the builtin datetime
arithmetic is "naive".  The docs have always promised this:

"""
datetime2 = datetime1 + timedelta (1)
datetime2 = datetime1 - timedelta (2)

1) datetime2 is a duration of timedelta removed from datetime1, moving
forward in time if timedelta.days > 0, or backward if timedelta.days <
0. The result has the same tzinfo attribute as the input datetime, and
datetime2 - datetime1 == timedelta after. OverflowError is raised if
datetime2.year would be smaller than MINYEAR or larger than MAXYEAR.
Note that no time zone adjustments are done even if the input is an
aware object.

2) Computes the datetime2 such that datetime2 + timedelta ==
datetime1. As for addition, the result has the same tzinfo attribute
as the input datetime, and no time zone adjustments are done even if
the input is aware. This isn’t quite equivalent to datetime1 +
(-timedelta), because -timedelta in isolation can overflow in cases
where datetime1 - timedelta does not.
"""

>, and is ridiculously difficult to make happen in all cases that we want to
> support (Riyahd time zone and leap seconds not included).
> That IS the specific, concrete problem, and if you don't believe me, there
> is nothing I can do to convince you.

I apologize if I've come off as unduly critical - I truly have been
_only_ trying to find out what "the problem" is.  That helps!  Thank
you.  Note that I've had nothing to do with datetime (except