On Fri, Jan 15, 2021 at 10:53 AM Larry Hastings <la...@hastings.org
<mailto:la...@hastings.org>> wrote:
Sorry it took me 3+ days to reply--I had a lot to think about
here. But I have good things to report!
On 1/11/21 8:42 PM, Guido van Rossum wrote:
On Mon, Jan 11, 2021 at 1:20 PM Larry Hastings
<la...@hastings.org <mailto:la...@hastings.org>> wrote:
PEP 563 states:
For code that uses type hints, the
typing.get_type_hints(obj, globalns=None, localns=None)
function correctly evaluates expressions back from its
string form.
So, if you are passing in a localns argument that isn't None,
okay, but you're not using them "correctly" according to the
language. Also, this usage won't be compatible with static
type checkers.
I think you're misreading PEP 563 here. The mention of
globalns=None, localns=None refers to the fact that these
parameters have defaults, not that you must pass None. Note that
the next paragraph in that PEP mentions eval(ann, globals,
locals) -- it doesn't say eval(ann, {}, {}).
I think that's misleading, then. The passage is telling you how
to "correctly evaluate[s] expressions", and how I read it was,
it's telling me I have to supply globalns=None and localns=None
for it to work correctly--which, I had to discover on my own, were
the default values. I don't understand why PEP 563 feels
compelled to define a function that it's not introducing, and in
fact had already shipped with Python two versions ago.
I suppose PEP 563 is ambiguous because on the one hand global symbols
are the only things that work out of the box, on the other hand you
can make other things work by passing the right scope (and there's
lots of code now that does so), and on the third hand, it claims that
get_type_hints() adds the class scope, which nobody noticed or
implemented until this week (there's a PR, can't recall the number).
But I think all this is irrelevant given what comes below.
Later in that same section, PEP 563 points out a problem with
annotations that reference class-scoped variables, and claims
that the implementation would run into problems because methods
can't "see" the class scope. This is indeed a problem for PEP
563, but *you* can easily generate correct code, assuming the
containing class exists in the global scope (and your solution
requires that anyway). So in this case
```
class Outer:
class Inner:
...
def method(self, a: Inner, b: Outer) -> None:
...
```
The generated code for the `__annotations__` property could just
have a reference to `Outer.Inner` for such cases:
```
def __annotations__():
return {"a": Outer.Inner, "b": Outer, "return": None}
```
This suggestion was a revelation for me. Previously, a
combination of bad experiences early on when hacking on compile
and symtable, and my misunderstanding of exactly what was being
asserted in the November 2017 thread, led me to believe that all I
could support was globals. But I've been turning this over in my
head for several days now, and I suspect I can support... just
about anything.
I can name five name resolution scenarios I might encounter. I'll
discuss them below, in increasing order of difficulty.
*First* is references to globals / builtins. That's already
working, it's obvious how it works, and I need not elaborate further.
Yup.
*Second* is local variables in an enclosing function scope:
def outer_fn():
class C: pass
def inner_fn(a:C=None): pass
return inner_fn
As you pointed out elsewhere in un-quoted text, I could make the
annotation a closure, so it could retain a reference to the value
of (what is from its perspective) the free variable "C".
Yup.
*Third* is local variables in an enclosing class scope, as you
describe above:
class OuterCls:
class InnerCls:
def method(a:InnerCls=None): pass
If I understand what you're suggesting, I could notice inside the
compiler that Inner is being defined in a class scope, walk up the
enclosing scopes until I hit the outermost class, then reconstruct
the chain of pulling out attributes until it resolves globally.
Thus I'd rewrite this example to:
class OuterCls:
class InnerCls:
def method(a:OuterCls.InnerCls=None): pass
We've turned the local reference into a global reference, and we
already know globals work fine.
I think this is going too far. A static method defined in InnerCls
does not see InnerCls (even after the class definitions are complete).
E.g.
```
class Outer:
class Inner:
@staticmethod
def foo(): return Inner
```
If you then call Outer.Inner.foo() you get "NameError: name 'Inner' is
not defined".
*Fourth* is local variables in an enclosing class scope, which are
themselves local variables in an enclosing function scope:
def outerfn():
class OuterCls:
class InnerCls:
def method(a:InnerCls=None): pass
return OuterCls.InnerCls
Even this is solvable, I just need to combine the "second" and
"third" approaches above. I walk up the enclosing scopes to find
the outermost class scope, and if that's a function scope, I
create a closure and retain a reference to /that/ free variable.
Thus this would turn into
def outerfn():
class OuterCls:
class InnerCls:
def method(a:OuterCls.InnerCls=None): pass
and method.__co_annotations__ would reference the free variable
"OuterCls" defined in outerfn.
Probably also not needed.
*Fifth* is the nasty one. Note that so far every definition we've
referred to in an annotation has been /before/ the definition of
the annotation. What if we want to refer to something defined
/after/ the annotation?
def outerfn():
class OuterCls:
class InnerCls:
def method(a:zebra=None): pass
...
We haven't seen the definition of "zebra" yet, so we don't know
what approach to take. It could be any of the previous four
scenarios. What do we do?
If you agree with me that (3) and (4) are unnecessary (or even
undesirable), the options here are either that zebra is a local in
outerfn() (then just make it a closure), and if it isn't you should
treat it as a global.
This is solvable too: we simply delay the compilation of
__co_annotations__ code objects until the very last possible
moment. First, at the time we bind the class or function, we
generate a stub __co_annotations__ object, just to give the
compiler what it expects. The compiler inserts it into the const
table for the enclosing construct (function / class / module), and
we remember what index it went into. Then, after we've finished
processing the entire AST tree for this module, but before we we
exit the compiler, we reconstruct the required context for
evaluating each __co_annotations__ function--the nested chain of
symbol tables, the compiler blocks if needed, etc--and evaluate
the annotations for real. We assemble the correct
__co_annotations__ code object and overwrite the stub in the const
table with this now-correct value.
I can't think of any more scenarios. So, I think I can handle
basically anything!
However, there are two scenarios where the behavior of evaluations
will change in a way the user might find surprising. The first is
when they redefine a variable used in an annotation:
x = str
def fn(a:x="345"): pass
x = int
With stock semantics, the annotation to "a" will be "str". With
PEP 563 or my PEP, the annotation to "a" will be "int". (It gets
even more exciting if you said "del x".)
This falls under the Garbage in, Garbage out principle. Mypy doesn't
even let you do this. Another type checker which is easy to install,
pyright, treats it as str. I wouldn't worry too much about it. If you
strike the first definition of x, the pyright complains and mypy
treats it as int.
Similarly, delaying the annotations so that we make everything
visible means defining variables with the same name in multiple
scopes may lead to surprising behavior.
x = str
class Outer:
def method(a:x="345"): pass
x = int
Again, stock gets you an annotation of "str", but PEP 563 and my
PEP gets you "str", because they'll see the /final/ result of
evaluating the body of Outer.
Sadly this is the price you pay for delayed evaluation of
annotations. Delaying the evaluation of annotations is the goal,
and the whole point is to make changes, observable by the user, in
how annotations are evaluated. All we can do is document these
behaviors and hope our users forgive us.
Agreed.
I think this is a vast improvement over the first draft of my PEP,
and assuming nobody points out major flaws in this approach (and,
preferably, at least a little encouragement), I plan to redesign
my prototype along these lines. (Though not right away--I want to
take a break and attend to some other projects first.)
Thanks for the mind-blowing suggestions, Guido! I must say,
you're pretty good at this Python stuff.
You're not so bad yourself -- without your wakeup call we would have
immortalized PEP 563's limitations.
--
--Guido van Rossum (python.org/~guido <http://python.org/~guido>)
/Pronouns: he/him //(why is my pronoun here?)/
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
