Re: [Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8

[Nick Coghlan]

On 5 September 2016 at 15:59, Steve Dower  wrote:
> +continue to default to ``locale.getpreferredencoding()`` (for text files) or
> +plain bytes (for binary files). This only affects the encoding used when 
> users
> +pass a bytes object to Python where it is then passed to the operating 
> system as
> +a path name.

For the three non-filesystem cases:

I checked the situation for os.environb, and that's already
unavailable on Windows (since os.supports_bytes_environ is False
there), while sys.argv is apparently already handled correctly (i.e.
always using the *W APIs).

That means my only open question would be the handling of subprocess
module calls (both with and without shell=True), since that currently
works with binary arguments on *nix:

>>> subprocess.call([b"python", b"-c", "print('ℙƴ☂ℌøἤ')".encode("utf-8")])
ℙƴ☂ℌøἤ
0
>>> subprocess.call(b"python -c '%s'" % 'print("ℙƴ☂ℌøἤ")'.encode("utf-8"), 
>>> shell=True)
ℙƴ☂ℌøἤ
0

While calling system native apps that way will still have many
portability challenges, there are also plenty of cases where folks use
sys.executable to launch new Python processes in a separate instance
of the currently running interpreter, and it would be good if these
changes brought cross-platform consistency to the handling of binary
arguments here as well.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 5:21 AM, Nick Coghlan  wrote:
> On 5 September 2016 at 04:40, Koos Zevenhoven  wrote:
>> On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi  wrote:
>>> On 4 September 2016 at 19:59, Nick Coghlan  wrote:
>> [...]

 Similarly, it would be reasonable to say that these three snippets
 should all be equivalent from a typechecking perspective:

 x = None # type: Optional[T]

 x: Optional[T] = None

 x: Optional[T]
 x = None
>>>
>>>
>>> Nice idea, explicit is better than implicit.
>>
>> How is it going to help that these are equivalent within one checker,
>> if the meaning may differ across checkers?
>
> For typechecker developers, it provides absolute clarity that the
> semantics of the new annotations should match the behaviour of
> existing type comments when there's an initialiser present,

I understood that, but what's the benefit? I hope there will be a type
checker that breaks this "rule".

> or of a
> parameter annotation when there's no initialiser present.

No, your suggested addition does not provide any reference to this.
(...luckily, because that would have been worse.)

> For folks following along without necessarily keeping up with all the
> nuances, it makes it more explicit what Guido means when he says "PEP
> 526 does not make a stand on the
> behavior of type checkers (other than deferring to PEP 484)."

What you are proposing is exactly "making a stand on the behavior of
type checkers", and the examples you provide below are all variations
of the same situation and provide no justification for a general rule.

Here's a general rule:

The closer it gets to the end of drafting a PEP [1],
the more carefully you have to justify changes.

Justification is left as an exercise ;-).

--Koos

[1] or any document (or anything, I suppose)

> For example, the formulation of the divergent initialisation case
> where I think the preferred semantics are already implied by PEP 484
> can be looked at this way:
>
> x = None # type: Optional[List[T]]
> if arg is not None:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
>
> It would be a strange typechecker indeed that handled that case
> differently from the new spellings made possible by PEP 526:
>
> x: Optional[List[T]] = None
> if arg is not None:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
>
> x: Optional[List[T]]
> if arg is None:
> x = None
> else:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
>
> x: Optional[List[T]]
> if arg is not None:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
> else:
> x = None
>
> Or from the semantics of PEP 484 parameter annotations:
>
> def my_func(arg:Optional[List[T]], other_arg=None):
> # other_arg is implicitly Optional[Any]
> if arg is not None and other_arg is not None:
> # Here, "arg" can be assumed to be List[T]
> # while "other_arg" is Any
> arg.extend(other_arg)
>
> A self-consistent typechecker will either allow all of the above, or
> prohibit all of the above, while a typechecker that *isn't*
> self-consistent would be incredibly hard to use.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Paul Moore]

On 5 September 2016 at 06:54, Steve Dower  wrote:
> +Using the raw object with small buffers
> +---
> +
> +Code that uses the raw IO object and attempts to read less than four 
> characters
> +will now receive an error. Because it's possible that any single character 
> may
> +require up to four bytes when represented in utf-8, requests must fail.

I'm very concerned about this statement. It's clearly not true that
the request *must* fail, as reading 1 byte from a UTF-8 enabled Linux
console stream currently works (at least I believe it does). And there
is code in the wild that works by doing a test that "there's input
available" (using kbhit on Windows and select on Unix) and then doing
read(1) to ensure a non-blocking read (the pyinvoke code I referenced
earlier). If we're going to break this behaviour, I'd argue that we
need to provide a working alternative.

At a minimum, can the PEP include a recommended cross-platform means
of implementing a non-blocking read from standard input, to replace
the current approach? (If the recommendation is to read a larger
4-byte buffer and manage the process of retaining unused bytes
yourself, then that's quite a major change to at least the code I'm
thinking of in invoke, and I'm not sure read(4) guarantees that it
*won't* block if only 1 byte is available without blocking...)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Martin Panter]

On 5 September 2016 at 09:10, Paul Moore  wrote:
> On 5 September 2016 at 06:54, Steve Dower  wrote:
>> +Using the raw object with small buffers
>> +---
>> +
>> +Code that uses the raw IO object and attempts to read less than four 
>> characters
>> +will now receive an error. Because it's possible that any single character 
>> may
>> +require up to four bytes when represented in utf-8, requests must fail.
>
> I'm very concerned about this statement. It's clearly not true that
> the request *must* fail, as reading 1 byte from a UTF-8 enabled Linux
> console stream currently works (at least I believe it does). And there
> is code in the wild that works by doing a test that "there's input
> available" (using kbhit on Windows and select on Unix) and then doing
> read(1) to ensure a non-blocking read (the pyinvoke code I referenced
> earlier). If we're going to break this behaviour, I'd argue that we
> need to provide a working alternative.
>
> At a minimum, can the PEP include a recommended cross-platform means
> of implementing a non-blocking read from standard input, to replace
> the current approach? (If the recommendation is to read a larger
> 4-byte buffer and manage the process of retaining unused bytes
> yourself, then that's quite a major change to at least the code I'm
> thinking of in invoke, and I'm not sure read(4) guarantees that it
> *won't* block if only 1 byte is available without blocking...)

FWIW, on Linux and Unix in general, if select() or similar indicates
that some read data is available, calling raw read() with any buffer
size should return at least one byte, whatever is available, without
blocking. If the user has only typed one byte, read(4) would return
that one byte immediately.

But if you are using a BufferedReader (stdin.buffer rather than
stdin.buffer.raw), then this guarantee is off and read(4) will block
until it gets 4 bytes, or until EOF.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 5 September 2016 at 18:19, Koos Zevenhoven  wrote:
> On Mon, Sep 5, 2016 at 5:21 AM, Nick Coghlan  wrote:
>> On 5 September 2016 at 04:40, Koos Zevenhoven  wrote:
>>> On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi  
>>> wrote:
 On 4 September 2016 at 19:59, Nick Coghlan  wrote:
>>> [...]
>
> Similarly, it would be reasonable to say that these three snippets
> should all be equivalent from a typechecking perspective:
>
> x = None # type: Optional[T]
>
> x: Optional[T] = None
>
> x: Optional[T]
> x = None


 Nice idea, explicit is better than implicit.
>>>
>>> How is it going to help that these are equivalent within one checker,
>>> if the meaning may differ across checkers?
>>
>> For typechecker developers, it provides absolute clarity that the
>> semantics of the new annotations should match the behaviour of
>> existing type comments when there's an initialiser present,
>
> I understood that, but what's the benefit? I hope there will be a type
> checker that breaks this "rule".

Such a typechecker means you're not writing Python anymore, you're
writing Java/C++/C# in a language that isn't designed to be used that
way.

Fortunately, none of the current typecheckers have made that mistake,
nor does anyone appear to be promoting this mindset outside this
particular discussion.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Paul Moore]

On 5 September 2016 at 10:37, Martin Panter  wrote:
> On 5 September 2016 at 09:10, Paul Moore  wrote:
>> On 5 September 2016 at 06:54, Steve Dower  wrote:
>>> +Using the raw object with small buffers
>>> +---
>>> +
>>> +Code that uses the raw IO object and attempts to read less than four 
>>> characters
>>> +will now receive an error. Because it's possible that any single character 
>>> may
>>> +require up to four bytes when represented in utf-8, requests must fail.
>>
>> I'm very concerned about this statement. It's clearly not true that
>> the request *must* fail, as reading 1 byte from a UTF-8 enabled Linux
>> console stream currently works (at least I believe it does). And there
>> is code in the wild that works by doing a test that "there's input
>> available" (using kbhit on Windows and select on Unix) and then doing
>> read(1) to ensure a non-blocking read (the pyinvoke code I referenced
>> earlier). If we're going to break this behaviour, I'd argue that we
>> need to provide a working alternative.
>>
>> At a minimum, can the PEP include a recommended cross-platform means
>> of implementing a non-blocking read from standard input, to replace
>> the current approach? (If the recommendation is to read a larger
>> 4-byte buffer and manage the process of retaining unused bytes
>> yourself, then that's quite a major change to at least the code I'm
>> thinking of in invoke, and I'm not sure read(4) guarantees that it
>> *won't* block if only 1 byte is available without blocking...)
>
> FWIW, on Linux and Unix in general, if select() or similar indicates
> that some read data is available, calling raw read() with any buffer
> size should return at least one byte, whatever is available, without
> blocking. If the user has only typed one byte, read(4) would return
> that one byte immediately.
>
> But if you are using a BufferedReader (stdin.buffer rather than
> stdin.buffer.raw), then this guarantee is off and read(4) will block
> until it gets 4 bytes, or until EOF.

OK. So a correct non-blocking approach would be:

def ready_for_reading():
if sys.platform == "win32":
return msvcrt.kbhit()
else:
reads, _, _ = select.select([sys.stdin], [], [], 0.0)
return bool(reads and reads[0] is sys.stdin)

if ready_for_reading():
return sys.stdin.buffer.raw.read(4)

And using a buffer any less than 4 bytes long risks an error on input
(specifically, if a character than encodes to multiple UTF-8 bytes is
returned).

OK. That's viable, I guess, although the *actual* code in question is
written to be valid on Python back to 2.7, and to work for general
file-like objects, so it'll still be some work to get the incantation
correct.

It would be nice to explain this explicitly in the docs, though, as
read(1) is pretty common, and doesn't typically expect to get an error
because of this.

Thanks,
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 1:04 PM, Nick Coghlan  wrote:
> On 5 September 2016 at 18:19, Koos Zevenhoven  wrote:
>> On Mon, Sep 5, 2016 at 5:21 AM, Nick Coghlan  wrote:
>>> On 5 September 2016 at 04:40, Koos Zevenhoven  wrote:
 On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi  
 wrote:
> On 4 September 2016 at 19:59, Nick Coghlan  wrote:
 [...]
>>
>> Similarly, it would be reasonable to say that these three snippets
>> should all be equivalent from a typechecking perspective:
>>
>> x = None # type: Optional[T]
>>
>> x: Optional[T] = None
>>
>> x: Optional[T]
>> x = None
>
>
> Nice idea, explicit is better than implicit.

 How is it going to help that these are equivalent within one checker,
 if the meaning may differ across checkers?
>>>
>>> For typechecker developers, it provides absolute clarity that the
>>> semantics of the new annotations should match the behaviour of
>>> existing type comments when there's an initialiser present,
>>
>> I understood that, but what's the benefit? I hope there will be a type
>> checker that breaks this "rule".
>
> Such a typechecker means you're not writing Python anymore, you're
> writing Java/C++/C# in a language that isn't designed to be used that
> way.

I'm glad those are all the languages you accuse me of. The list could
have been a lot worse. I actually have some good memories of Java. It
felt kind of cool at that age, and it taught me many things about
undertanding the structure of large and complicated programs after I
had been programming for years in other languages, including C++. It
also taught me to value simplicity instead, so here we are.

> Fortunately, none of the current typecheckers have made that mistake,
> nor does anyone appear to be promoting this mindset outside this
> particular discussion.

The thing I'm promoting here is to not add anything to PEP 526 that
says what a type checker is supposed to do with type annotations.
Quite the opposite of Java/C++/C#, I would say.

We can, of course, speculate about the future of type checkers and the
implications of PEP 526 on it. That's what I'm trying to do on
python-ideas, speculate about the best kind of type checking
(achievable with PEP 526 annotations) [1].

--Koos


[1] https://mail.python.org/pipermail/python-ideas/2016-September/042076.html

>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

On Mon, Sep 05, 2016 at 11:19:38AM +0300, Koos Zevenhoven wrote:
> On Mon, Sep 5, 2016 at 5:21 AM, Nick Coghlan  wrote:
[...]
> > On 5 September 2016 at 04:40, Koos Zevenhoven  wrote:
> >> On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi  
> >> wrote:
> >>> On 4 September 2016 at 19:59, Nick Coghlan  wrote:
> >> [...]


[Ivan Levkivskyi]
>  Similarly, it would be reasonable to say that these three snippets
>  should all be equivalent from a typechecking perspective:
> 
>  x = None # type: Optional[T]
> 
>  x: Optional[T] = None
> 
>  x: Optional[T]
>  x = None
[...]

[Koos Zevenhoven]
> >> How is it going to help that these are equivalent within one checker,
> >> if the meaning may differ across checkers?

Before I can give an answer to your [Koos'] question, I have to 
understand what you see as the problem here.

I *think* that you are worried that two different checkers will disagree 
on what counts as a type error. That given the same chunk of code:

x: Optional[T] = None
if x:
spam(x)
else:
   x.eggs()

two checkers will disagree as to whether or not the code is safe. Is 
that your concern? If not, can you explain in more detail what your 
concern is?


[Nick Coghlan]
> > For typechecker developers, it provides absolute clarity that the
> > semantics of the new annotations should match the behaviour of
> > existing type comments when there's an initialiser present,

[Koos]
> I understood that, but what's the benefit? 

Are you asking what is the benefit of having three forms of syntax for 
the same thing?

The type comment systax is required for Python 2 and backwards- 
compatibility. That's a given.

The variable annotation syntax is required because the type comment 
syntax is (according to the PEP) very much a second-best solution. See 
the PEP:

https://www.python.org/dev/peps/pep-0526/#id4

So this is a proposal to create a *better* syntax for something which 
already exists. The old version, using comments, cannot be deprecated or 
removed, as it is required for Python 3.5 and older.

Once we allow 

x: T = value

then there benefit in also allowing:

x: T
x = value

since this supports some of the use cases that aren't well supported by 
type comments or one-line variable annotations. E.g. very long or deeply 
indented lines, situations where the assignment to x is inside an 
if...else branch, or any other time you wish to declare the type of the 
variable before actually setting the variable.



[Koos]
> I hope there will be a type checker that breaks this "rule".

I don't understand. Do you mean that you want three different behaviours 
for these type annotations? What would they do differently?

To me, all three are clear and obvious ways of declaring the type of a 
variable. Whether I write `x: T = expr` or `x = expr  #type:T`, it 
should be clear that I intend `x` to be treated as T. What would you do 
differently?


[Nick]
> > or of a
> > parameter annotation when there's no initialiser present.

[Koos] 
> No, your suggested addition does not provide any reference to this.
> (...luckily, because that would have been worse.)

I'm sorry, I don't follow you. Are you suggesting that we should have 
the syntax `name:T = value` mean something different inside and outside 
of a function parameter list?

def func(x:T = v):
y:T = v


The first line declares x as type T with default value v; the second 
line declares y as type T with initial value v. You say this is 
"worse"... worse than what? What behaviour would you prefer to see?



[Nick]
> > For folks following along without necessarily keeping up with all the
> > nuances, it makes it more explicit what Guido means when he says "PEP
> > 526 does not make a stand on the
> > behavior of type checkers (other than deferring to PEP 484)."

[Koos] 
> What you are proposing is exactly "making a stand on the behavior of
> type checkers", and the examples you provide below are all variations
> of the same situation and provide no justification for a general rule.

I'm sorry, I don't understand this objection. The closest I can get to 
an answer would be:

A general rule is better than a large number of unconnected, arbitrary, 
special cases.

Does that help?


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Steve Dower]

The best fix is to use a buffered reader, which will read all the available 
bytes and then let you .read(1), even if it happens to be an incomplete 
character.

We could theoretically add buffering to the raw reader to handle one character, 
which would allow very small reads from raw, but that severely complicates 
things and the advice to use a buffered reader is good advice anyway.

Top-posted from my Windows Phone

-Original Message-
From: "Paul Moore" 
Sent: ‎9/‎5/‎2016 3:23
To: "Martin Panter" 
Cc: "Python Dev" 
Subject: Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

On 5 September 2016 at 10:37, Martin Panter  wrote:
> On 5 September 2016 at 09:10, Paul Moore  wrote:
>> On 5 September 2016 at 06:54, Steve Dower  wrote:
>>> +Using the raw object with small buffers
>>> +---
>>> +
>>> +Code that uses the raw IO object and attempts to read less than four 
>>> characters
>>> +will now receive an error. Because it's possible that any single character 
>>> may
>>> +require up to four bytes when represented in utf-8, requests must fail.
>>
>> I'm very concerned about this statement. It's clearly not true that
>> the request *must* fail, as reading 1 byte from a UTF-8 enabled Linux
>> console stream currently works (at least I believe it does). And there
>> is code in the wild that works by doing a test that "there's input
>> available" (using kbhit on Windows and select on Unix) and then doing
>> read(1) to ensure a non-blocking read (the pyinvoke code I referenced
>> earlier). If we're going to break this behaviour, I'd argue that we
>> need to provide a working alternative.
>>
>> At a minimum, can the PEP include a recommended cross-platform means
>> of implementing a non-blocking read from standard input, to replace
>> the current approach? (If the recommendation is to read a larger
>> 4-byte buffer and manage the process of retaining unused bytes
>> yourself, then that's quite a major change to at least the code I'm
>> thinking of in invoke, and I'm not sure read(4) guarantees that it
>> *won't* block if only 1 byte is available without blocking...)
>
> FWIW, on Linux and Unix in general, if select() or similar indicates
> that some read data is available, calling raw read() with any buffer
> size should return at least one byte, whatever is available, without
> blocking. If the user has only typed one byte, read(4) would return
> that one byte immediately.
>
> But if you are using a BufferedReader (stdin.buffer rather than
> stdin.buffer.raw), then this guarantee is off and read(4) will block
> until it gets 4 bytes, or until EOF.

OK. So a correct non-blocking approach would be:

def ready_for_reading():
if sys.platform == "win32":
return msvcrt.kbhit()
else:
reads, _, _ = select.select([sys.stdin], [], [], 0.0)
return bool(reads and reads[0] is sys.stdin)

if ready_for_reading():
return sys.stdin.buffer.raw.read(4)

And using a buffer any less than 4 bytes long risks an error on input
(specifically, if a character than encodes to multiple UTF-8 bytes is
returned).

OK. That's viable, I guess, although the *actual* code in question is
written to be valid on Python back to 2.7, and to work for general
file-like objects, so it'll still be some work to get the incantation
correct.

It would be nice to explain this explicitly in the docs, though, as
read(1) is pretty common, and doesn't typically expect to get an error
because of this.

Thanks,
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

It looks like you are trying to make sense of this, but unfortunately
there's some added mess and copy&paste-like errors regarding who said
what. I think no such errors remain in what I quote below:

On Mon, Sep 5, 2016 at 3:10 PM, Steven D'Aprano  wrote:
>
> [Koos Zevenhoven]
>> >> How is it going to help that these are equivalent within one checker,
>> >> if the meaning may differ across checkers?
>
> Before I can give an answer to your [Koos'] question, I have to
> understand what you see as the problem here.

The problem was that suggested restrictive addition into PEP 526 with
no proper justification, especially since the PEP was not supposed to
restrict the semantics of type checking. I was asking how it would
help to add that restriction. Very simple. Maybe some people got
confused because I did want to *discuss* best practices for type
checking elsewhere.

> I *think* that you are worried that two different checkers will disagree
> on what counts as a type error. That given the same chunk of code:

In the long term, I'm worried about that, but there's nothing that PEP
526 can do about it at this point.

> [Nick Coghlan]
>> > For typechecker developers, it provides absolute clarity that the
>> > semantics of the new annotations should match the behaviour of
>> > existing type comments when there's an initialiser present,
>
> [Koos]
>> I understood that, but what's the benefit?
>
> Are you asking what is the benefit of having three forms of syntax for
> the same thing?

No, still the same thing: What is the benefit of that particular
restriction, when there are no other restrictions? Better just leave
it out.

> The type comment systax is required for Python 2 and backwards-
> compatibility. That's a given.

Sure, but all type checkers will not have to care about Python 2.

> The variable annotation syntax is required because the type comment
> syntax is (according to the PEP) very much a second-best solution. See
> the PEP:
>
> https://www.python.org/dev/peps/pep-0526/#id4
>
> So this is a proposal to create a *better* syntax for something which
> already exists. The old version, using comments, cannot be deprecated or
> removed, as it is required for Python 3.5 and older.

Right.

> Once we allow
>
> x: T = value
>
> then there benefit in also allowing:
>
> x: T
> x = value
>
> since this supports some of the use cases that aren't well supported by
> type comments or one-line variable annotations. E.g. very long or deeply
> indented lines, situations where the assignment to x is inside an
> if...else branch, or any other time you wish to declare the type of the
> variable before actually setting the variable.

Sure.

> [Nick]
>> > For folks following along without necessarily keeping up with all the
>> > nuances, it makes it more explicit what Guido means when he says "PEP
>> > 526 does not make a stand on the
>> > behavior of type checkers (other than deferring to PEP 484)."
>
> [Koos]
>> What you are proposing is exactly "making a stand on the behavior of
>> type checkers", and the examples you provide below are all variations
>> of the same situation and provide no justification for a general rule.
>
> I'm sorry, I don't understand this objection. The closest I can get to
> an answer would be:
>
> A general rule is better than a large number of unconnected, arbitrary,
> special cases.

A general rule that does not solve a problem is worse than no rule.


-- Koos


>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 5 September 2016 at 21:46, Koos Zevenhoven  wrote:
> The thing I'm promoting here is to not add anything to PEP 526 that
> says what a type checker is supposed to do with type annotations.

PEP 526 says it doesn't intend to expand the scope of typechecking
semantics beyond what PEP 484 already supports. For that to be true,
it needs to be able to define expected equivalencies between the
existing semantics of PEP 484 and the new syntax in PEP 526.

If those equivalencies can't be defined, then Mark's concerns are
valid, and the PEP either needs to be deferred as inadvertently
introducing new semantics while intending to only introduce new
syntax, or else the intended semantics need to be spelled out as they
were in PEP 484 so folks can judge the proposal accurately, rather
than attempting to judge it based on an invalid premise.

For initialised variables, the equivalence between the two PEPs is
straightforward: "x: T = expr" is equivalent to "x = expr # type: T"

If PEP 526 always required an initialiser, and didn't introduce
ClassVar, there'd be no controversy, and we'd already be done.

However, the question of "Does this new syntax necessarily imply the
introduction of new semantics?" gets a lot murkier for uninitialised
variables.

A strict "no new semantics beyond PEP 484" interpretation would mean
that these need to be interpreted the same way as parameter
annotations: as a type hint on the outcome of the code executed up to
that point, rather than as a type constraint on assignment statements
in the code *following* that point.

Consider:

def simple_appender(base: List[T], value: T) -> None:
base.append(value)

This will typecheck fine - lists have append methods, and the value
appended conforms to what our list expects.

The parameter annotations mainly act as constraints on how this
function is *called*, with the following all being problematic:

simple_appender([1, 2, 3], "hello") # Container/value type mismatch
simple_appender([1, 2, 3], None) # Value is not optional
simple_appender((1, 2, 3), 4) # A tuple is not a list

However, because of the way name binding in Python works, the
annotations in *no way* constrain assignments inside the function
body:

def not_so_simple_appender(base: List[T], value: T) -> None:
other_ref = base
base = value
other_ref.append(base)

>From a dynamic typechecking perspective, that's just as valid as the
original implementation, since the "List[T]" type of "other_ref" is
inferred from the original type of "base" before it gets rebound to
value and has its new type inferred as "T".

This freedom to rebind an annotated name without a typechecker
complaining is what Mark is referring to when he says that PEP 484
attaches annotations to expressions rather than types.

Under such "parameter annotation like" semantics, uninitialised
variable annotations would only make sense as a new form of
post-initialisation assertion, and perhaps as some form of
Eiffel-style class invariant documentation syntax.

The usage to help ensure code correctness in multi-branch
initialisation cases would then look something like this:

   if case1:
x = ...
elif case2:
x = ...
else:
x = ...
assert x : List[T] # If we get to here without x being List[T],
something's wrong

The interpreter could then optimise type assertions out entirely at
function level (even in __debug__ mode), and turn them into
annotations at module and class level (with typecheckers then deciding
how to process them).

That's not what the PEP proposes for uninitialised variables though:
it proposes processing them *before* a series of assignment
statements, which *only makes sense* if you plan to use them to
constrain those assignments in some way.

If you wanted to write something like that under a type assertion
spelling, then you could enlist the aid of the "all" builtin:

assert all(x) : List[T] # All local assignments to "x" must abide
by this constraint
if case1:
x = ...
elif case2:
x = ...
else:
x = ...

So I've come around to the point of view of being a solid -1 on the
PEP as written - despite the best of intentions, it strongly
encourages "assert all(x): List[T]" as the default interpretation of
unitialised variable annotations, and doesn't provide an easy way to
do arbitrary inline type assertions to statically check the
correctness of the preceding code the way we can with runtime
assertions and as would happen if the code in question was factored
out to an annotated function.

Stick the "assert" keyword in front of them though, call them type
assertions rather than type declarations, and require all() when you
want to constrain all assignments later in the function (or until the
next relevant type assertion), and I'm a solid +1.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing l

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

Sorry, I don't have time to read emails of this length now, and
perhaps I'm interpreting your emails more literally than you write
them, anyway.

If PEP 484 introduces unnecessary restrictions at this point, that's a
separate issue. I see no need to copy those into PEP 526. I'll be
posting my own remaining concerns regarding PEP 526 when I find the
time.

-- Koos


On Mon, Sep 5, 2016 at 4:46 PM, Nick Coghlan  wrote:
> On 5 September 2016 at 21:46, Koos Zevenhoven  wrote:
>> The thing I'm promoting here is to not add anything to PEP 526 that
>> says what a type checker is supposed to do with type annotations.
>
> PEP 526 says it doesn't intend to expand the scope of typechecking
> semantics beyond what PEP 484 already supports. For that to be true,
> it needs to be able to define expected equivalencies between the
> existing semantics of PEP 484 and the new syntax in PEP 526.
>
> If those equivalencies can't be defined, then Mark's concerns are
> valid, and the PEP either needs to be deferred as inadvertently
> introducing new semantics while intending to only introduce new
> syntax, or else the intended semantics need to be spelled out as they
> were in PEP 484 so folks can judge the proposal accurately, rather
> than attempting to judge it based on an invalid premise.
>
> For initialised variables, the equivalence between the two PEPs is
> straightforward: "x: T = expr" is equivalent to "x = expr # type: T"
>
> If PEP 526 always required an initialiser, and didn't introduce
> ClassVar, there'd be no controversy, and we'd already be done.
>
> However, the question of "Does this new syntax necessarily imply the
> introduction of new semantics?" gets a lot murkier for uninitialised
> variables.
>
> A strict "no new semantics beyond PEP 484" interpretation would mean
> that these need to be interpreted the same way as parameter
> annotations: as a type hint on the outcome of the code executed up to
> that point, rather than as a type constraint on assignment statements
> in the code *following* that point.
>
> Consider:
>
> def simple_appender(base: List[T], value: T) -> None:
> base.append(value)
>
> This will typecheck fine - lists have append methods, and the value
> appended conforms to what our list expects.
>
> The parameter annotations mainly act as constraints on how this
> function is *called*, with the following all being problematic:
>
> simple_appender([1, 2, 3], "hello") # Container/value type mismatch
> simple_appender([1, 2, 3], None) # Value is not optional
> simple_appender((1, 2, 3), 4) # A tuple is not a list
>
> However, because of the way name binding in Python works, the
> annotations in *no way* constrain assignments inside the function
> body:
>
> def not_so_simple_appender(base: List[T], value: T) -> None:
> other_ref = base
> base = value
> other_ref.append(base)
>
> From a dynamic typechecking perspective, that's just as valid as the
> original implementation, since the "List[T]" type of "other_ref" is
> inferred from the original type of "base" before it gets rebound to
> value and has its new type inferred as "T".
>
> This freedom to rebind an annotated name without a typechecker
> complaining is what Mark is referring to when he says that PEP 484
> attaches annotations to expressions rather than types.
>
> Under such "parameter annotation like" semantics, uninitialised
> variable annotations would only make sense as a new form of
> post-initialisation assertion, and perhaps as some form of
> Eiffel-style class invariant documentation syntax.
>
> The usage to help ensure code correctness in multi-branch
> initialisation cases would then look something like this:
>
>if case1:
> x = ...
> elif case2:
> x = ...
> else:
> x = ...
> assert x : List[T] # If we get to here without x being List[T],
> something's wrong
>
> The interpreter could then optimise type assertions out entirely at
> function level (even in __debug__ mode), and turn them into
> annotations at module and class level (with typecheckers then deciding
> how to process them).
>
> That's not what the PEP proposes for uninitialised variables though:
> it proposes processing them *before* a series of assignment
> statements, which *only makes sense* if you plan to use them to
> constrain those assignments in some way.
>
> If you wanted to write something like that under a type assertion
> spelling, then you could enlist the aid of the "all" builtin:
>
> assert all(x) : List[T] # All local assignments to "x" must abide
> by this constraint
> if case1:
> x = ...
> elif case2:
> x = ...
> else:
> x = ...
>
> So I've come around to the point of view of being a solid -1 on the
> PEP as written - despite the best of intentions, it strongly
> encourages "assert all(x): List[T]" as the default interpretation of
> unitialised variable annotations, and doesn't provide an easy way to
> do arbitrary inli

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 5 September 2016 at 23:46, Nick Coghlan  wrote:
> Under such "parameter annotation like" semantics, uninitialised
> variable annotations would only make sense as a new form of
> post-initialisation assertion, and perhaps as some form of
> Eiffel-style class invariant documentation syntax.

Thinking further about the latter half of that comment, I realised
that the PEP 484 equivalence I'd like to see for variable annotations
in a class body is how they would relate to a property definition
using the existing PEP 484 syntax.

For example, consider:

class AnnotatedProperty:

@property
def x(self) -> int:
...

@x.setter
def x(self, value: int) -> None:
...

@x.deleter
def x(self) -> None:
...

It would be rather surprising if that typechecked differently from:

class AnnotatedVariable:

x: int

For ClassVar, you'd similarly want:


class AnnotatedClassVariable:

x: ClassVar[int]

to typecheck like "x" was declared as an annotated property on the metaclass.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Mark Shannon]

On 04/09/16 21:16, Guido van Rossum wrote:

Everybody please stop panicking. PEP 526 does not make a stand on the
behavior of type checkers (other than deferring to PEP 484). If you
want to start a discussion about constraining type checkers please do
it over at python-ideas. There is no rush as type checkers are not
affected by the feature freeze.



Indeed, we shouldn't panic. We should take our time, review this 
carefully and make sure that the version of typehints that lands in 3.7 
is one that we most of us are happy with and all of us can at least 
tolerate.


Cheers,
Mark.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 5:02 PM, Nick Coghlan  wrote:
> On 5 September 2016 at 23:46, Nick Coghlan  wrote:
>> Under such "parameter annotation like" semantics, uninitialised
>> variable annotations would only make sense as a new form of
>> post-initialisation assertion,

Why not discuss this in the python-ideas thread where I quote myself
from last Friday regarding the notion of annotations as assertions?

>> and perhaps as some form of
>> Eiffel-style class invariant documentation syntax.

I hope this is simpler than it sounds :-)

> Thinking further about the latter half of that comment, I realised
> that the PEP 484 equivalence I'd like to see for variable annotations
> in a class body is how they would relate to a property definition
> using the existing PEP 484 syntax.
>
> For example, consider:
>
> class AnnotatedProperty:
>
> @property
> def x(self) -> int:
> ...
>
> @x.setter
> def x(self, value: int) -> None:
> ...
>
> @x.deleter
> def x(self) -> None:
> ...
>
> It would be rather surprising if that typechecked differently from:
>
> class AnnotatedVariable:
>
> x: int
>

How about just using the latter way? That's much clearer. I doubt this
needs a change in the PEP.

> For ClassVar, you'd similarly want:
>
>
> class AnnotatedClassVariable:
>
> x: ClassVar[int]
>
> to typecheck like "x" was declared as an annotated property on the metaclass.
>

Sure, there are many things that one may consider equivalent. I doubt
you'll be able to list them all in a way that everyone agrees on. And
I hope you don't take this as a challenge -- I'm in the don't-panic
camp :).


-- Koos


> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Wolfgang]

Hi,

first if something like this is needed I am fine with the syntax.

But I think this changes comes to late for 3.6. There is more time needed
to discuss all of this and there is more time needed to mature the type
checkers. Don't rush with such a change because it affects the language at 
whole.
So please defer it.

Saying the syntax is fine means not I am happy with the addition.
Fundamentally I think don't add this at Python language syntax level.
Years come and it will be misused misunderstood by new users.
It will affect all other users reading the code and even misguiding them.

If Variable and Attribute Annotation is needed keep it simply at the
stub file level on *.pyi files. Only there for the type checking stuff.
Other users must not bother with them.

And for stub files it can be as simple as:

myvar = typing.Int()

or all other valid syntax.


For me the whole specifying of types in Python comes down to:
It can be useful to document a user interface (most of the time a function
or method) and say if you call it these types are supported.

At some day a documentation generator can use this type information
and I have no longer the need to specify it also in the docstring.

Personally I would like to extend the pyi stub files to carry also the
documentation and keep the code as clean and short as possible.
Sometimes the documentation is longer than the code and the code is no longer
easy to find.
Instead of putting everything into the language put more into the stub files.
Even micropython or other implementations with limited constraints don't need
to carry all of this. Even if it is only part of the AST, it is overhead.


Have someone checked if there is a possibility if this is added to slow
down the interpreter or interpreter startup or increase the memory
consumption?


Regards,

Wolfgang


On 30.08.2016 23:20, Guido van Rossum wrote:
> I'm happy to present PEP 526 for your collective review:
> https://www.python.org/dev/peps/pep-0526/ (HTML)
> https://github.com/python/peps/blob/master/pep-0526.txt (source)
> 
> There's also an implementation ready:
> https://github.com/ilevkivskyi/cpython/tree/pep-526
> 
> I don't want to post the full text here but I encourage feedback on
> the high-order ideas, including but not limited to
> 
> - Whether (given PEP 484's relative success) it's worth adding syntax
> for variable/attribute annotations.
> 
> - Whether the keyword-free syntax idea proposed here is best:
>   NAME: TYPE
>   TARGET: TYPE = VALUE
> 
> Note that there's an extensive list of rejected ideas in the PEP;
> please be so kind to read it before posting here:
> https://www.python.org/dev/peps/pep-0526/#rejected-proposals-and-things-left-out-for-now
> 
> 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Wolfgang]

Hi,

first if something like this is needed I am fine with the syntax.

But I think this changes comes to late for 3.6. There is more time needed
to discuss all of this and there is more time needed to mature the type
checkers. Don't rush with such a change because it affects the language at 
whole.
So please defer it.

Saying the syntax is fine means not I am happy with the addition.
Fundamentally I think don't add this at Python language syntax level.
Years come and it will be misused misunderstood by new users.
It will affect all other users reading the code and even misguiding them.

If Variable and Attribute Annotation is needed keep it simply at the
stub file level on *.pyi files. Only there for the type checking stuff.
Other users must not bother with them.

And for stub files it can be as simple as:

myvar = typing.Int()

or all other valid syntax.


For me the whole specifying of types in Python comes down to:
It can be useful to document a user interface (most of the time a function
or method) and say if you call it these types are supported.

At some day a documentation generator can use this type information
and I have no longer the need to specify it also in the docstring.

Personally I would like to extend the pyi stub files to carry also the
documentation and keep the code as clean and short as possible.
Sometimes the documentation is longer than the code and the code is no longer
easy to find.
Instead of putting everything into the language put more into the stub files.
Even micropython or other implementations with limited constraints don't need
to carry all of this. Even if it is only part of the AST, it is overhead.


Have someone checked if there is a possibility if this is added to slow
down the interpreter or interpreter startup or increase the memory
consumption?


Regards,

Wolfgang


On 30.08.2016 23:20, Guido van Rossum wrote:
> I'm happy to present PEP 526 for your collective review:
> https://www.python.org/dev/peps/pep-0526/ (HTML)
> https://github.com/python/peps/blob/master/pep-0526.txt (source)
> 
> There's also an implementation ready:
> https://github.com/ilevkivskyi/cpython/tree/pep-526
> 
> I don't want to post the full text here but I encourage feedback on
> the high-order ideas, including but not limited to
> 
> - Whether (given PEP 484's relative success) it's worth adding syntax
> for variable/attribute annotations.
> 
> - Whether the keyword-free syntax idea proposed here is best:
>   NAME: TYPE
>   TARGET: TYPE = VALUE
> 
> Note that there's an extensive list of rejected ideas in the PEP;
> please be so kind to read it before posting here:
> https://www.python.org/dev/peps/pep-0526/#rejected-proposals-and-things-left-out-for-now
> 
> 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Do PEP 526 type declarations define the types of variables or not?

[Mark Shannon]

Hi,

PEP 526 states that "This PEP aims at adding syntax to Python for 
annotating the types of variables" and Guido seems quite insistent that 
the declarations are for the types of variables.


However, I get the impression that most (all) of the authors and 
proponents of PEP 526 are quite keen to emphasise that the PEP in no way 
limits type checkers from doing what they want.


This is rather contradictory. The behaviour of a typechecker is defined 
by the typesystem that it implements. Whether a type annotation 
determines the type of a variable or an expression alters changes what 
typesystems are feasible. So, stating that annotations define the type 
of variables *does* limit what a typechecker can or cannot do.


Unless of course, others may have a different idea of what the "type of 
a variable" means.

To me, it means it means that for all assignments `var = expr`
the type of `expr` must be a subtype of the variable,
and for all uses of var, the type of the use is the same as the type of 
the variable.


In this example:

def bar()->Optional[int]: ...

def foo()->int:
x:Optional[int] = bar()
if x is None:
return -1
return x

According to PEP 526 the annotation `x:Optional[int]`
means that the *variable* `x` has the type `Optional[int]`.
So what is the type of `x` in `return x`?
If it is `Optional[int]`, then a type checker is obliged to reject this 
code. If it is `int` then what does "type of a variable" actually mean,

and why aren't the other uses of `x` int as well?

Cheers,
Mark.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

On Mon, Sep 5, 2016 at 7:19 AM, Mark Shannon  wrote:
> On 04/09/16 21:16, Guido van Rossum wrote:
>>
>> Everybody please stop panicking. PEP 526 does not make a stand on the
>> behavior of type checkers (other than deferring to PEP 484). If you
>> want to start a discussion about constraining type checkers please do
>> it over at python-ideas. There is no rush as type checkers are not
>> affected by the feature freeze.
>>
>
> Indeed, we shouldn't panic. We should take our time, review this carefully
> and make sure that the version of typehints that lands in 3.7 is one that we
> most of us are happy with and all of us can at least tolerate.

Right, we want the best possible version to land in 3.7. And in order
to make that possible, I have to accept it *provisionally* for 3.6 and
Ivan's implementation will go into 3.6b1. We will then have until 3.7
to experiment with it and tweak it as necessary.

Maybe ClassVar will turn out to be pointless. Maybe we'll decide that
we want to have a syntax for quickly annotating several variables with
the same type (x, y, z: T). Maybe we'll change the rules for how or
when __annotations__ is updated. Maybe we'll change slightly whether
we'll allow annotating complex assignment targets like x[f()].

But without starting the experiment now we won't be able to evaluate
any of those things. Waiting until 3.7 is just going to cause the
exact same discussions that are going on now 18 months from now.

Regarding how type checkers should use the new syntax, PEP 526 itself
give barely more guidance than PEP 3107, except that we now have PEP
484 to tell us what types ought to look like, *if* you want to use an
external type checker.

I hope that you and others will help write another PEP
(informational?) to guide type checkers and their users. Given my own
experience at Dropbox (much of it vicariously through the eyes of the
many Dropbox engineers annotating their own code) I am *very*
reluctant to try and specify the behavior of a type checker formally
myself. As anyone who has used mypy on a sizeable project knows, there
are a lot more details to sort out than how to handle branches that
assign different values to the same variable.

For people who want to read about what it is like to use mypy
seriously, I can recommend the series of three blog posts by Daniel
Moisset starting here:
http://www.machinalis.com/blog/a-day-with-mypy-part-1/

If you want to see a large open source code base that's annotated for
mypy (with 97% coverage), I recommend looking at Zulip:
https://github.com/zulip/zulip
Try digging through the history and looking for commits mentioning
mypy; a Google Summer of Code student did most of the work over the
summer. (The syntax used is the Python-2-compatible version, but
that's hardly relevant -- the important things to observe include how
they use types and how they had to change their code to pacify mypy.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do PEP 526 type declarations define the types of variables or not?

[Ryan Gonzalez]

Maybe the PEP should just say it's for "annotating variables", and it would
mention "primarily for the purpose of types"?

--
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something’s wrong.
http://kirbyfan64.github.io/
On Sep 5, 2016 10:27 AM, "Mark Shannon"  wrote:

> Hi,
>
> PEP 526 states that "This PEP aims at adding syntax to Python for
> annotating the types of variables" and Guido seems quite insistent that the
> declarations are for the types of variables.
>
> However, I get the impression that most (all) of the authors and
> proponents of PEP 526 are quite keen to emphasise that the PEP in no way
> limits type checkers from doing what they want.
>
> This is rather contradictory. The behaviour of a typechecker is defined by
> the typesystem that it implements. Whether a type annotation determines the
> type of a variable or an expression alters changes what typesystems are
> feasible. So, stating that annotations define the type of variables *does*
> limit what a typechecker can or cannot do.
>
> Unless of course, others may have a different idea of what the "type of a
> variable" means.
> To me, it means it means that for all assignments `var = expr`
> the type of `expr` must be a subtype of the variable,
> and for all uses of var, the type of the use is the same as the type of
> the variable.
>
> In this example:
>
> def bar()->Optional[int]: ...
>
> def foo()->int:
> x:Optional[int] = bar()
> if x is None:
> return -1
> return x
>
> According to PEP 526 the annotation `x:Optional[int]`
> means that the *variable* `x` has the type `Optional[int]`.
> So what is the type of `x` in `return x`?
> If it is `Optional[int]`, then a type checker is obliged to reject this
> code. If it is `int` then what does "type of a variable" actually mean,
> and why aren't the other uses of `x` int as well?
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%
> 40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

On Mon, Sep 05, 2016 at 04:40:08PM +0300, Koos Zevenhoven wrote:

> On Mon, Sep 5, 2016 at 3:10 PM, Steven D'Aprano  wrote:
> >
> > [Koos Zevenhoven]
> >> >> How is it going to help that these are equivalent within one checker,
> >> >> if the meaning may differ across checkers?
> >
> > Before I can give an answer to your [Koos'] question, I have to
> > understand what you see as the problem here.
> 
> The problem was that suggested restrictive addition into PEP 526 with
> no proper justification, especially since the PEP was not supposed to
> restrict the semantics of type checking. 

What "suggested restrictive addition into PEP 526" are you referring to?

Please be specific.


> I was asking how it would
> help to add that restriction. Very simple. Maybe some people got
> confused because I did want to *discuss* best practices for type
> checking elsewhere.

I still can't answer your question, because I don't understand what 
restriction you are talking about. Unless you mean the restriction that 
variable annotations are to mean the same thing whether they are written 
as `x:T = v` or `x = v #type: T`. I don't see this as a restriction.



> > The type comment systax is required for Python 2 and backwards-
> > compatibility. That's a given.
> 
> Sure, but all type checkers will not have to care about Python 2.

They will have to care about type comments until such time as they are 
ready to abandon all versions of Python older than 3.6.

And even then, there will probably be code still written with type 
comments until Python 4000.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Chris Angelico]

On Tue, Sep 6, 2016 at 2:17 AM, Steven D'Aprano  wrote:
>> > The type comment systax is required for Python 2 and backwards-
>> > compatibility. That's a given.
>>
>> Sure, but all type checkers will not have to care about Python 2.
>
> They will have to care about type comments until such time as they are
> ready to abandon all versions of Python older than 3.6.

More specifically, until *the code they check* can abandon all <3.6.
If the checker itself depends on new features (say, an improved AST
parser that retains inline comments for subsequent evaluation), you
could say "You must have Python 3.6 or better to use this checker",
but the application itself would still be able to run on older
versions. That's another reason not to delay this PEP until 3.7, as
it'd push _everything_ another 18 months (or more) into the future.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Paul Moore]

On 5 September 2016 at 14:36, Steve Dower  wrote:
> The best fix is to use a buffered reader, which will read all the available
> bytes and then let you .read(1), even if it happens to be an incomplete
> character.

But this is sys.stdin.buffer.raw, we're talking about. People can't
really layer anything on top of that, it's precisely because they are
trying to *bypass* the existing layering (that doesn't work the way
that they need it to, because it blocks) that is the problem here.

> We could theoretically add buffering to the raw reader to handle one 
> character,
> which would allow very small reads from raw, but that severely complicates
> things and the advice to use a buffered reader is good advice anyway.

Can you provide an example of how I'd rewrite the code that I quoted
previously to follow this advice? Note - this is not theoretical, I
expect to have to provide a PR to fix exactly this code should this
change go in. At the moment I can't find a way that doesn't impact the
(currently working and not expected to need any change) Unix version
of the code, most likely I'll have to add buffering of 4-byte reads
(which as you say is complex).

The problem I have is that we're forcing application code to do the
buffering to cater for Windows (where you're proposing that the raw IO
layer doesn't handle it and will potentially fail reads of <4 bytes).
Code written for POSIX doesn't need to do that, and the additional
maintenance overhead is potentially large enough to put POSIX
developers off adding the necessary code - this is in direct contrast
to the proposal to make fsencoding UTF-8 to make it easier for
POSIX-compatible code to "just work" on Windows.

If the goals are to handle Unicode correctly for stdin, and to work in
a way that POSIX-compatible code works without special effort on
Windows, then as far as I can see we have to handle the buffering of
partial reads of UTF-8 code sequences (because POSIX does so). If, on
the other hand, we just want Unicode to work on Windows, and we're not
looking for POSIX code to work without change, then the proposed
behaviour is OK (although I still maintain it needs to be flagged, as
it's very close to being a compatibility break in practice, even if
it's technically within the rules).

Paul

PS I'm not 100% sure that under POSIX read() will return partial UTF-8
byte sequences. I think it must, because otherwise a lot of code I've
seen would be broken, but if a POSIX expert can confirm or deny my
assumption, that would be great.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Ethan Furman]

On 09/03/2016 09:48 AM, Nick Coghlan wrote:

On 3 September 2016 at 21:35, Martin Panter wrote:

On 3 September 2016 at 08:47, Victor Stinner wrote:

Le samedi 3 septembre 2016, Random832 a écrit :

On Fri, Sep 2, 2016, at 19:44, Ethan Furman wrote:



The problem with only having `bchr` is that it doesn't help with
`bytearray`;


What is the use case for bytearray.fromord? Even in the rare case
someone needs it, why not bytearray(bchr(...))?


Yes, this was my point: I don't think that we need a bytearray method to
create a mutable string from a single byte.


I agree with the above. Having an easy way to turn an int into a bytes
object is good. But I think the built-in bchr() function on its own is
enough. Just like we have bytes object literals, but the closest we
have for a bytearray literal is bytearray(b". . .").


This is a good point - earlier versions of the PEP didn't include
bchr(), they just had the class methods, so "bytearray(bchr(...))"
wasn't an available spelling (if I remember the original API design
correctly, it would have been something like
"bytearray(bytes.byte(...))"), which meant there was a strong
consistency argument in having the alternate constructor on both
types. Now that the PEP proposes the "bchr" builtin, the "fromord"
constructors look less necessary.


tl;dr -- Sounds good to me.  I'll update the PEP.

---

When this started the idea behind the methods that eventually came to be
called "fromord" and "fromsize" was that they would be the two possible
interpretations of "bytes(x)":

  the legacy Python2 behavior:

>>> var = bytes('abc')
>>> bytes(var[1])
'b'

  the current Python 3 behavior:

>>> var = b'abc'
>>> bytes(var[1])
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
  \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
  \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
  \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
  \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
  \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
  \x00\x00'

Digging deeper the problem turns out to be that indexing a bytes object
changed:

  Python 2:

>>> b'abc'[1]
'b'

  Python 3:

>>> b'abc'[1]
98

If we pass an actual byte into the Python 3 bytes constructor it behaves
as one would expect:

>>> bytes(b'b')
b'b'

Given all this it can be argued that the real problem is that indexing a
bytes object behaves differently depending on whether you retrieve a single
byte with an index versus a single byte with a slice:

>>> b'abc'[2]
99

>>> b'abc'[2:]
b'c'

Since we cannot fix that behavior, the question is how do we make it more
livable?

- we can add a built-in to transform the int back into a byte:

  >>> bchr(b'abc'[2])
  b'c'

- we can add a method to return a byte from the bytes object, not an int:

  >>> b'abc'.getbyte(2)
  b'c'

- we can add a method to return a byte from an int:

  >>> bytes.fromint(b'abc'[2])
  b'c'

Which is all to say we have two problems to deal with:

- getting bytes from a bytes object
- getting bytes from an int

Since "bytes.fromint()" and "bchr()" are the same, and given that
"bchr(ordinal)" mirrors "chr(ordinal)", I think "bchr" is the better
choice for getting bytes from an int.

For getting bytes from bytes, "getbyte()" and "iterbytes" are good choices.


Given that, and the uncertain deprecation time frame for accepting
integers in the main bytes and bytearray constructors, perhaps both
the "fromsize" and "fromord" parts of the proposal can be deferred
indefinitely in favour of just adding the bchr() builtin?


Agreed.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ethan Furman]

On 09/05/2016 06:46 AM, Nick Coghlan wrote:

[an easy to understand explanation for those of us who aren't type-inferring 
gurus]

Thanks, Nick.  I think I finally have a grip on what Mark was talking about, 
and about how these things should work.

Much appreciated!

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Steve Dower]

On 05Sep2016 0941, Paul Moore wrote:

On 5 September 2016 at 14:36, Steve Dower  wrote:

The best fix is to use a buffered reader, which will read all the available
bytes and then let you .read(1), even if it happens to be an incomplete
character.


But this is sys.stdin.buffer.raw, we're talking about. People can't
really layer anything on top of that, it's precisely because they are
trying to *bypass* the existing layering (that doesn't work the way
that they need it to, because it blocks) that is the problem here.


This layer also blocks, and always has. You need to go to platform 
specific functions anyway to get non-blocking functionality (which is 
also wrapped up in getc I believe, but that isn't used by FileIO or the 
new WinConsoleIO classes).



We could theoretically add buffering to the raw reader to handle one character,
which would allow very small reads from raw, but that severely complicates
things and the advice to use a buffered reader is good advice anyway.


Can you provide an example of how I'd rewrite the code that I quoted
previously to follow this advice? Note - this is not theoretical, I
expect to have to provide a PR to fix exactly this code should this
change go in. At the moment I can't find a way that doesn't impact the
(currently working and not expected to need any change) Unix version
of the code, most likely I'll have to add buffering of 4-byte reads
(which as you say is complex).


The easiest way to follow it is to use "sys.stdin.buffer.read(1)" rather 
than "sys.stdin.buffer.raw.read(1)".



PS I'm not 100% sure that under POSIX read() will return partial UTF-8
byte sequences. I think it must, because otherwise a lot of code I've
seen would be broken, but if a POSIX expert can confirm or deny my
assumption, that would be great.


I just tested, and yes it returns partial characters. That's a good 
reason to do the single character buffering ourselves. Shouldn't be too 
hard to deal with.


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do PEP 526 type declarations define the types of variables or not?

[Guido van Rossum]

On Mon, Sep 5, 2016 at 8:26 AM, Mark Shannon  wrote:
> PEP 526 states that "This PEP aims at adding syntax to Python for annotating
> the types of variables" and Guido seems quite insistent that the
> declarations are for the types of variables.
>
> However, I get the impression that most (all) of the authors and proponents
> of PEP 526 are quite keen to emphasise that the PEP in no way limits type
> checkers from doing what they want.
>
> This is rather contradictory. The behaviour of a typechecker is defined by
> the typesystem that it implements. Whether a type annotation determines the
> type of a variable or an expression alters changes what typesystems are
> feasible. So, stating that annotations define the type of variables *does*
> limit what a typechecker can or cannot do.
>
> Unless of course, others may have a different idea of what the "type of a
> variable" means.
> To me, it means it means that for all assignments `var = expr`
> the type of `expr` must be a subtype of the variable,
> and for all uses of var, the type of the use is the same as the type of the
> variable.
>
> In this example:
>
> def bar()->Optional[int]: ...
>
> def foo()->int:
> x:Optional[int] = bar()
> if x is None:
> return -1
> return x
>
> According to PEP 526 the annotation `x:Optional[int]`
> means that the *variable* `x` has the type `Optional[int]`.
> So what is the type of `x` in `return x`?
> If it is `Optional[int]`, then a type checker is obliged to reject this
> code. If it is `int` then what does "type of a variable" actually mean,
> and why aren't the other uses of `x` int as well?

Oh, there is definitely a problem here if you interpret it that way.
Of course I assume that other type checkers are at least as smart as
mypy. :-) In mypy, the analysis of this example narrows the type x can
have once `x is None` is determined to be false, so that the example
passes.

I guess this is a surprise if you think of type systems like Java's
where the compiler forgets what it has learned, at least from the
language spec's POV. But a Python type checker is more like a linter,
and false positives (complaints about valid code) are much more
problematic than false negatives (passing invalid code).

So a Python type checker that is to gain acceptance of users must be
much smarter than that, and neither PEP 484 not PEP 526 is meant to
require a type checker to complain about `return x` in the above
example.

I'm not sure how to change the language of the PEP though -- do you
have a suggestion? It all seems to depend on how the reader interprets
the meaning of very vague words like "variable" and "type".

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do PEP 526 type declarations define the types of variables or not?

[Steven D'Aprano]

On Mon, Sep 05, 2016 at 04:26:17PM +0100, Mark Shannon wrote:

> In this example:
> 
> def bar()->Optional[int]: ...
> 
> def foo()->int:
> x:Optional[int] = bar()
> if x is None:
> return -1
> return x
>
> According to PEP 526 the annotation `x:Optional[int]`
> means that the *variable* `x` has the type `Optional[int]`.

We can change that to read:

x = bar()

and let the type-checker infer the type of x. Introducing the annotation 
here is a red-herring: you have *exactly* the same issue whether we do 
type inference, a type comment, or the proposed variable annotation.



> So what is the type of `x` in `return x`?

The type of *the variable x* is still Optional[int]. But that's the 
wrong question.

The right question is, what's the type of the return result?

The return result is not "the variable x". The return result is the 
value produced by evaluating the expression `x` in the specific context 
of where the return statement is found.

(To be precise, it is the *inferred* return value, of course, since the 
actual return value won't be produced until runtime.)


> If it is `Optional[int]`, then a type checker is obliged to reject this 
> code. 

Not at all, because the function isn't returning "the variable x". It's 
returning the value currently bound to x, and *that* is known to be an 
int. It has to be an int, because if it were None, the function would 
have already returned -1.

The return result is an expression that happens to consist of just a 
single term, in this case `x`. To make it more clear, let's change it 
to `return x+999`.

The checker should be able to infer that since `x` must be an int here, 
the expression `x+999` will also be an int. This satisfies the return 
type. Of course `x+999` is just a stand-in for any expression that is 
known to return an int, and that includes the case where the expression 
is `x` alone.

There's really not anything more mysterious going on here than the case 
where we have a Union type with two branches that depend on which type 
x actually is:


def demo(x:Union[int, str])->int:
# the next two lines are expected to fail the type check
# since the checker can't tell if x is an int or a str
x+1
len(x)
# but the rest of the function should pass
if isinstance(x, int):
# here we know x is definitely an int
y = x + 1
if isinstance(x, str):
# and here we know x is definitely a str
y = len(x)
return y



When I run MyPy on that, it gives:

[steve@ando ~]$ mypy test.py
test.py: note: In function "demo":
test.py:6: error: Unsupported operand types for + ("Union[int, str]" and "int")
test.py:7: error: Argument 1 to "len" has incompatible type "Union[int, str]"; 
expected "Sized"


But all of this is a red herring. It has nothing to do with the proposed 
variable annotation syntax: it applies equally to type comments and 
function annotations.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Paul Moore]

On 5 September 2016 at 18:38, Steve Dower  wrote:
>> Can you provide an example of how I'd rewrite the code that I quoted
>> previously to follow this advice? Note - this is not theoretical, I
>> expect to have to provide a PR to fix exactly this code should this
>> change go in. At the moment I can't find a way that doesn't impact the
>> (currently working and not expected to need any change) Unix version
>> of the code, most likely I'll have to add buffering of 4-byte reads
>> (which as you say is complex).
>
> The easiest way to follow it is to use "sys.stdin.buffer.read(1)" rather
> than "sys.stdin.buffer.raw.read(1)".

I may have got confused here. If I say sys.stdin.buffer.read(1),
having first checked via kbhit() that there's a character
available[1], then I will always get 1 byte returned, never the
"buffer too small to return a full character" error that you talk
about in the PEP? If so, then I don't understand when the error you
propose will be raised (unless your comment here is based on what you
say below that we'll now buffer and therefore the error is no longer
needed).

>
>> PS I'm not 100% sure that under POSIX read() will return partial UTF-8
>> byte sequences. I think it must, because otherwise a lot of code I've
>> seen would be broken, but if a POSIX expert can confirm or deny my
>> assumption, that would be great.
>
> I just tested, and yes it returns partial characters. That's a good reason
> to do the single character buffering ourselves. Shouldn't be too hard to
> deal with.

OK, cool. Again I'm slightly confused because isn't this what you said
before "severely complicates things" - or was that only for the raw
layer?

I was over-simplifying the issue for pyinvoke, which in practice is
complicated by not yet having completed the process disentangling
bytes/unicode handling. As a result, I was handwaving somewhat about
whether read() is called on a raw stream or a buffered stream - in
practice I'm sure I can manage as long as the buffered level still
handles single-byte reads.

One thing I did think of, though - if someone *is* working at the raw
IO level, they have to be prepared for the new "buffer too small to
return a full character" error. That's OK. But what if they request
reading 7 bytes, but the input consists of 6 character s that encode
to 1 byte in UTF-8, followed by a character that encodes to 2 bytes?
You can return 6 bytes, that's fine - but you'll presumably still need
to read the extra character before you can determine that it won't fit
- so you're still going to have to buffer to some degree, surely? I
guess this is implementation details, though - I'll try to find some
time to read the patch in order to understand this. It's not something
that matters in terms of the PEP anyway, it's an implementation
detail.

Cheers,
Paul

[1] Yes, I know that's not the best approach, but it's as good as we
get without adding rather too much scary Windows specific code. (The
irony of trying to get this right given how much low-level Unix code I
don't follow is already in there doesn't escape me :-()
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

On Mon, Sep 5, 2016 at 10:24 AM, Ethan Furman  wrote:
> On 09/05/2016 06:46 AM, Nick Coghlan wrote:
>
> [an easy to understand explanation for those of us who aren't type-inferring
> gurus]
>
> Thanks, Nick.  I think I finally have a grip on what Mark was talking about,
> and about how these things should work.
>
> Much appreciated!

There must be some misunderstanding. The message from Nick with that
timestamp 
(https://mail.python.org/pipermail/python-dev/2016-September/146200.html)
hinges on an incorrect understanding of the intention of annotations
without value (e.g. `x: Optional[int]`), leading to a -1 on the PEP.

I can't tell if this is an honest misunderstanding or a strawman, but
I want to set the intention straight.

First of all, the PEP does not require the type checker to interpret
anything in a particular way; it intentionally shies away from
prescribing semantics (other than the runtime semantics of updating
__annotations__ or verifying that the target appears assignable).

But there appears considerable fear about what expectations the PEP
has of a reasonable type checker. In response to this I'll try to
sketch how I think this should be implemented in mypy.

There are actually at least two separate cases: if x is a local
variable, the intention of `x: ` is quite different from when x
occurs in a class.

- When found in a class, all *uses* (which may appear in modules far
away from the definition) must be considered to conform to the stated
type -- as must all assignments to it, but I believe that's never been
in doubt. There are just too many edge cases to consider to make
stricter assumptions (e.g. threading, exceptions, signals), so that
even after seeing `self.x = 42; use(self.x)` the call to use() cannot
assume that self.x is still 42.

- But when found inside a function referring to a local variable, mypy
should treat the annotation as a restriction on assignment, and use
its own inference engine to type-check *uses* of that variable. So
that in this example (after Mark's):

def bar() -> Optional[int]: ...

def foo() -> int:
x: Optional[int]
x = bar()
if x is None:
return -1
return x

there should not be an error on `return x` because mypy is smart
enough to know it cannot be None at that point.

I am at a loss how to modify the PEP to avoid this misunderstanding,
since it appears it is entirely in the reader's mind. The PEP is not a
tutorial but a spec for the implementation, and as a spec it is quite
clear that it leaves the type-checking semantics up to individual type
checkers. And I think that is the right thing to do -- in practice
there are many other ways to write the above example, and mypy will
understand some of them, but not others, while other type checkers may
understand a different subset of examples. I can't possibly prescribe
how type checkers should behave in each case -- I can't even tell
which cases are important to distinguish.

So writing down "the type checker should not report an error in the
following case" in the PEP is not going to be helpful for anyone (in
contrast, I think discussing examples on a mailing list *is* useful).
Like a linter, a type checker has limited intelligence, and it will be
a quality of implementation issue as to how useful a type checker will
be in practice. But that's not the topic of PEP 526.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Push PEP 528 (use utf8 on Windows) right now, but revert before 3.6 if needed

[Victor Stinner]

Hi,

I just spoke with Steve Dower (thanks for the current sprint!) about
the PEP 528. We somehow agreed that we need to push his implementation
of the PEP right now to get enough time to test as much applications
as possible on Windows to have a wide view of possible all
regressions.

The hope is that enough users will test the first Python 3.6 beta 1
(feature freeze!) with their app on Windows.

If we find blocker points before Python 3.6 final, we still have time
to revert code to restore the Python 3.5 behaviour (use ANSI code page
for bytes).

What do you think?

Note: First I was strongly opposed to any kind of change related to
encodings on Windows, but then I made my own tests and had to confess
that my proposed changes break the world. Steve's approach makes more
sense and is more realistic.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Terry Reedy]

On 9/5/2016 11:34 AM, Guido van Rossum wrote:

On Mon, Sep 5, 2016 at 7:19 AM, Mark Shannon  wrote:



Indeed, we shouldn't panic. We should take our time, review this carefully
and make sure that the version of typehints that lands in 3.7 is one that we
most of us are happy with and all of us can at least tolerate.


Right, we want the best possible version to land in 3.7. And in order
to make that possible, I have to accept it *provisionally* for 3.6 and


Until now, the 'provisional' part has not been clear to me, and 
presumably others who have written as if acceptance meant 'baked in 
stone'.  We have had provisional modules, but not, that I can think of, 
syntax that remains provisional past the x.y.0 release.



Ivan's implementation will go into 3.6b1. We will then have until 3.7
to experiment with it and tweak it as necessary.


New syntax is usually implemented within python itself, and can be fully 
experimented with during alpha and beta releases.  In this case, the 
effective implementation will be in 3rd party checkers and 
experimentation will take longer.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ivan Levkivskyi]

On 5 September 2016 at 20:15, Guido van Rossum  wrote:

> There are actually at least two separate cases: if x is a local
> variable, the intention of `x: ` is quite different from when x
> occurs in a class.
>

If I understand you correctly this also matches my mental model.
In local scope

x: ann = value

acts like a filter allowing only something compatible to be assigned
at this point (and/or casting to a more precise type).
While in class or module it is a part of an "API specification" for that
class/module.


> I am at a loss how to modify the PEP to avoid this misunderstanding,
> since it appears it is entirely in the reader's mind. The PEP is not a
> tutorial but a spec for the implementation, ...
>

I was thinking about changing terminology to name annotations, but that will
not solve problem. The PEP mentions a separate document (guidelines) that
will be published.
I think a real solution will be to make a separate PEP that will explain in
details
what is preferred meaning of types and what people and machines could do
with types.

Is anyone interested in going in this direction? I would like to especially
invite Mark,
you have a lot of experience with types inference that would be very helpful
(also it seems to me that you are concerned about this).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Push PEP 528 (use utf8 on Windows) right now, but revert before 3.6 if needed

[Ned Deily]

On 9/5/16 11:44, Victor Stinner wrote:
> I just spoke with Steve Dower (thanks for the current sprint!) about
> the PEP 528. We somehow agreed that we need to push his implementation
> of the PEP right now to get enough time to test as much applications
> as possible on Windows to have a wide view of possible all
> regressions.
> 
> The hope is that enough users will test the first Python 3.6 beta 1
> (feature freeze!) with their app on Windows.
> 
> If we find blocker points before Python 3.6 final, we still have time
> to revert code to restore the Python 3.5 behaviour (use ANSI code page
> for bytes).
> 
> What do you think?

Let's do it.

Thanks to both of you for hashing this out.




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Steve Dower]

On 05Sep2016 1110, Paul Moore wrote:

On 5 September 2016 at 18:38, Steve Dower  wrote:

Can you provide an example of how I'd rewrite the code that I quoted
previously to follow this advice? Note - this is not theoretical, I
expect to have to provide a PR to fix exactly this code should this
change go in. At the moment I can't find a way that doesn't impact the
(currently working and not expected to need any change) Unix version
of the code, most likely I'll have to add buffering of 4-byte reads
(which as you say is complex).


The easiest way to follow it is to use "sys.stdin.buffer.read(1)" rather
than "sys.stdin.buffer.raw.read(1)".


I may have got confused here. If I say sys.stdin.buffer.read(1),
having first checked via kbhit() that there's a character
available[1], then I will always get 1 byte returned, never the
"buffer too small to return a full character" error that you talk
about in the PEP? If so, then I don't understand when the error you
propose will be raised (unless your comment here is based on what you
say below that we'll now buffer and therefore the error is no longer
needed).


I don't think using buffer.read and kbhit together is going to be 
reliable anyway, as you may not have read everything that's already 
buffered yet. It's likely feasible if you flush everything, but 
otherwise it's a bit messy.



One thing I did think of, though - if someone *is* working at the raw
IO level, they have to be prepared for the new "buffer too small to
return a full character" error. That's OK. But what if they request
reading 7 bytes, but the input consists of 6 character s that encode
to 1 byte in UTF-8, followed by a character that encodes to 2 bytes?
You can return 6 bytes, that's fine - but you'll presumably still need
to read the extra character before you can determine that it won't fit
- so you're still going to have to buffer to some degree, surely? I
guess this is implementation details, though - I'll try to find some
time to read the patch in order to understand this. It's not something
that matters in terms of the PEP anyway, it's an implementation
detail.


If you do raw.read(7), we internally do "7 / 4" and decide to only read 
one wchar_t from the console. So the returned bytes will be between 1 
and 4 bytes long and there will be more info waiting for next time you ask.


The only case we can reasonably handle at the raw layer is "n / 4" is 
zero but n != 0, in which case we can read and cache up to 4 bytes (one 
wchar_t) and then return those in future calls. If we try to cache any 
more than that we're substituting for buffered reader, which I don't 
want to do.


Does caching up to one (Unicode) character at a time sound reasonable? I 
think that won't be much trouble, since there's no interference between 
system calls in that case and it will be consistent with POSIX behaviour.


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[eryk sun]

I have some suggestions. With ReadConsoleW, CPython can use the
pInputControl parameter to set a CtrlWakeup mask. This enables a
Unix-style Ctrl+D for ending a read without having to press enter. For
example:

>>> CTRL_MASK = 1 << 4
>>> inctrl = (ctypes.c_ulong * 4)(16, 0, CTRL_MASK, 0)
>>> _ = kernel32.ReadConsoleW(hStdIn, buf, 100, pn, inctrl); print()
spam
>>> buf.value
'spam\x04'
>>> pn[0]
5

read() would have to manually replace '\x04' with NUL. Ctrl+Z can also
be added to the mask:

>>> CTRL_MASK = 1 << 4 | 1 << 26
>>> inctrl = (ctypes.c_ulong * 4)(16, 0, CTRL_MASK, 0)
>>> _ = kernel32.ReadConsoleW(hStdIn, buf, 100, pn, inctrl); print()
spam
>>> buf.value
'spam\x1a'

I'd like a method to query, set and unset
ENABLE_VIRTUAL_TERMINAL_PROCESSING mode for the screen buffer
(sys.stdout and sys.stderr) without having to use ctypes. The console
in Windows 10 has built-in VT100 emulation, but it's initially
disabled. The cmd shell enables it, but Python scripts aren't always
run from cmd.exe. Sometimes they're run in a new console from Explorer
or via "start", etc. For example, IPython could check for this to
provide more bells and whistles when PyReadline isn't installed.

Finally, functions such as WriteConsoleInputW and
ReadConsoleOutputCharacter require opening CONIN$ or CONOUT$ with
GENERIC_READ | GENERIC_WRITE access. The initial handles given to a
console process have read-write access. For opening a new handle by
device name, WindowsConsoleIO should first try GENERIC_READ |
GENERIC_WRITE -- with a fallback to either GENERIC_READ or
GENERIC_WRITE. The fallback is necessary for CON, which uses the
desired access to determine whether to open the input buffer or screen
buffer.

---

Paul, do you have example code that uses the 'raw' stream? Using the
buffer should behave as it always has -- at least in this regard.
sys.stdin.buffer requests a large block, such as 8 KB. But since the
console defaults to a cooked mode (i.e. processed input and line input
-- control keys, command-line editing, input history, and aliases),
ReadConsole returns when enter is pressed or when interrupted. It
returns at least '\r\n', unless interrupted by Ctrl+C, Ctrl+Break or a
custom CtrlWakeup key. However, if line-input mode is disabled,
ReadConsole returns as soon as one or more characters is available in
the input buffer.

As to kbhit() returning true, this does not mean that read(1) from
console input won't block (not unless line-input mode is disabled). It
does mean that getwch() won't block (note the "w" in there; this one
reads Unicode characters).The CRT's conio functions (e.g. kbhit,
getwch) put the console input buffer in a raw mode (e.g. ^C is read as
'\x03' instead of generating a CTRL_C_EVENT) and call the lower-level
functions PeekConsoleInputW (kbhit) and ReadConsoleInputW (getwch), to
peek at and read input event records.

---

Splitting surrogate pairs across reads is a problem. Granted, this
should rarely be an issue given the size of the reads that the buffer
requests and the typical line length. In most cases the buffer
completely consumes the entire line in one read. But in principle the
raw stream shouldn't replace split surrogates with the U+FFFD
replacement character. For example, with Steve's patch from issue
1602:

>>> _ = write_console_input('\U0001\r\n');\
... b1 = raw_read(4); b2 = raw_read(4); b3 = raw_read(8)
𐀀
>>> b1, b2
(b'\xef\xbf\xbd', b'\xef\xbf\xbd')

Splitting UTF-8 sequences across writes is more common. Currently a
raw write doesn't handle this correctly:

>>> b = 'eggs \U0001 spam\n'.encode('utf-8')
>>> _ = raw_write(b[:6]); _ = raw_write(b[6:])
eggs  spam

Also, the console is UCS-2, which can't be transcoded between UTF-16
and UTF-8. Supporting UCS-2 in the console would integrate nicely with
the filesystem PEP. It makes it always possible to print
os.listdir('.'), copy and paste, and read it back without data loss.

It would probably be simpler to use UTF-16 in the main pipeline and
implement Martin's suggestion to mix in a UTF-8 buffer. The UTF-16
buffer could be renamed as "wbuffer", for expert use. However, if
you're fully committed to transcoding in the raw layer, I'm certain
that these problems can be addressed with small buffers and using
Python's codec machinery for a flexible mix of "surrogatepass" and
"replace" error handling.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Steve Dower]

On 05Sep2016 1234, eryk sun wrote:

Also, the console is UCS-2, which can't be transcoded between UTF-16
and UTF-8. Supporting UCS-2 in the console would integrate nicely with
the filesystem PEP. It makes it always possible to print
os.listdir('.'), copy and paste, and read it back without data loss.


Supporting UTF-8 actually works better for this. We already use 
surrogatepass explicitly (on the filesystem side, with PEP 529) and 
implicitly (on the console side, using the Windows conversion API).



It would probably be simpler to use UTF-16 in the main pipeline and
implement Martin's suggestion to mix in a UTF-8 buffer. The UTF-16
buffer could be renamed as "wbuffer", for expert use. However, if
you're fully committed to transcoding in the raw layer, I'm certain
that these problems can be addressed with small buffers and using
Python's codec machinery for a flexible mix of "surrogatepass" and
"replace" error handling.


I don't think it actually makes things simpler. Having two buffers is 
generally a bad idea unless they are perfectly synced, which would be 
impossible here without data corruption (if you read half a utf-8 
character sequence and then read the wide buffer, do you get that 
character or not?).


Writing a partial character is easily avoidable by the user. We can 
either fail with an error or print garbage, and currently printing 
garbage is the most compatible behaviour. (Also occurs on Linux - I have 
a VM running this week for testing this stuff.)


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] TLS handshake performance boost

[Christian Heimes]

Hi,

I have yet another patch for the ssl module,
http://bugs.python.org/issue19500 . The patch adds support for SSL
session resumption on the client side. A SSLContext automatically
handles server-side session. SSL sessions speed up successive TLS
connections to the same host considerable. My naïve benchmark shows
about 15 to 20% performance improvements for short-lived connections to
PyPI. In real-life applications with keep-alive, the speed-up will be a
bit smaller. Cory expects that requests is going to be about 5% faster
for subsequent requests.
https://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077.html
has more information on the topic.

Why is session handling different on the client side? OpenSSL does not
re-use sessions on the client side automatically. To use session
resumptions a SSL_SESSION must be copied from an established SSLSocket
to a new SSLSocket before the handshake. OpenSSL has further
restrictions, e.g. both sockets must use the same SSLContext. Session
cannot be shared between SSLContext.

My patch takes care of these details. The basic features are pretty much
done and tested. But I won't be able to write all documentation by the
end of the week or to write a high-level mechanism to auto-reuse
sessions. I still like to get the feature in before Monday.

What do you think? Are you fine with low-level session feature and
reduced documentation for the beta release?

Christian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Paul Moore]

On 5 September 2016 at 20:30, Steve Dower  wrote:
> The only case we can reasonably handle at the raw layer is "n / 4" is zero
> but n != 0, in which case we can read and cache up to 4 bytes (one wchar_t)
> and then return those in future calls. If we try to cache any more than that
> we're substituting for buffered reader, which I don't want to do.
>
> Does caching up to one (Unicode) character at a time sound reasonable? I
> think that won't be much trouble, since there's no interference between
> system calls in that case and it will be consistent with POSIX behaviour.

Caching a single character sounds perfectly OK. As I noted previously,
my use case probably won't need to work at the raw level anyway, so I
no longer expect to have code that will break, but I think that a
1-character buffer ensuring that we avoid surprises for code that was
written for POSIX is a good trade-off.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TLS handshake performance boost

[Ned Deily]

On 9/5/16 12:57, Christian Heimes wrote:
> I have yet another patch for the ssl module,
> http://bugs.python.org/issue19500 . The patch adds support for SSL
> session resumption on the client side. [...]
> 
> My patch takes care of these details. The basic features are pretty much
> done and tested. But I won't be able to write all documentation by the
> end of the week or to write a high-level mechanism to auto-reuse
> sessions. I still like to get the feature in before Monday.
> 
> What do you think? Are you fine with low-level session feature and
> reduced documentation for the beta release?

Unless there are other objections, I'm willing to make an exception if
you can get the current patch reviewed by one of the usual suspects from
the Security Sig, the missing pieces are in before b2, and the API of
the checked-in pieces doesn't change after b1.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Paul Moore]

On 5 September 2016 at 20:34, eryk sun  wrote:
> Paul, do you have example code that uses the 'raw' stream? Using the
> buffer should behave as it always has -- at least in this regard.
> sys.stdin.buffer requests a large block, such as 8 KB. But since the
> console defaults to a cooked mode (i.e. processed input and line input
> -- control keys, command-line editing, input history, and aliases),
> ReadConsole returns when enter is pressed or when interrupted. It
> returns at least '\r\n', unless interrupted by Ctrl+C, Ctrl+Break or a
> custom CtrlWakeup key. However, if line-input mode is disabled,
> ReadConsole returns as soon as one or more characters is available in
> the input buffer.

The code I'm looking at doesn't use the raw stream (I think). The
problem I had (and the reason I was concerned) is that the code does
some rather messy things, and without tracing back through the full
code path, I'm not 100% sure *what* level of stream it's using.
However, now that I know that the buffered layer won't ever error
because 1 byte isn't enough to return a full character, if I need to
change the code I can do so by switching to the buffered layer and
fixing the issue that way (although with Steve's new proposal even
that won't be necessary).

> As to kbhit() returning true, this does not mean that read(1) from
> console input won't block (not unless line-input mode is disabled). It
> does mean that getwch() won't block (note the "w" in there; this one
> reads Unicode characters).The CRT's conio functions (e.g. kbhit,
> getwch) put the console input buffer in a raw mode (e.g. ^C is read as
> '\x03' instead of generating a CTRL_C_EVENT) and call the lower-level
> functions PeekConsoleInputW (kbhit) and ReadConsoleInputW (getwch), to
> peek at and read input event records.

I understand. The code I'm working on was originally written for pure
POSIX, with all the termios calls to set the console into unbuffered
mode. In addition, it was until recently using the Python 2 text
model, and so there's a lot of places in the code where it's still
confused about whether it's processing bytes or characters (we've got
rid of a *lot* of "let's decode and see if that helps" calls...). At
the moment, kbhit(), while not correct, is "good enough". When I get
the time, and we get to a point where it's enough of a priority, I may
well look at refactoring this stuff to use proper Windows calls via
ctypes to do "read what's available". But that's a way off yet.

Thanks for the information, though, I'll keep it in mind when we do
get to a point where we're looking at this.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[eryk sun]

On Mon, Sep 5, 2016 at 7:54 PM, Steve Dower  wrote:
> On 05Sep2016 1234, eryk sun wrote:
>>
>> Also, the console is UCS-2, which can't be transcoded between UTF-16
>> and UTF-8. Supporting UCS-2 in the console would integrate nicely with
>> the filesystem PEP. It makes it always possible to print
>> os.listdir('.'), copy and paste, and read it back without data loss.
>
> Supporting UTF-8 actually works better for this. We already use
> surrogatepass explicitly (on the filesystem side, with PEP 529) and
> implicitly (on the console side, using the Windows conversion API).

CP_UTF8 requires valid UTF-16 text. MultiByteToWideChar and
WideCharToMultiByte are of no practical use here. For example:

>>> raw_read = sys.stdin.buffer.raw.read
>>> _ = write_console_input('\ud800\ud800\r\n'); raw_read(16)
��
b'\xef\xbf\xbd\xef\xbf\xbd\r\n'

This requires Python's "surrogatepass" error handler. It's also
required to decode UTF-8 that's potentially WTF-8 from
os.listdir(b'.'). Coming from the wild, there's a chance that
arbitrary bytes have invalid sequences other than lone surrogates, so
it needs to fall back on "replace" to deal with errors that
"surrogatepass" doesn't handle.

> Writing a partial character is easily avoidable by the user. We can either
> fail with an error or print garbage, and currently printing garbage is the
> most compatible behaviour. (Also occurs on Linux - I have a VM running this
> week for testing this stuff.)

Are you sure about that? The internal screen buffer of a Linux
terminal is bytes; it doesn't transcode to a wide-character format. In
the Unix world, almost everything is "get a byte, get a byte, get a
byte, byte, byte". Here's what I see in Ubuntu using GNOME Terminal,
for example:

>>> raw_write = sys.stdout.buffer.raw.write
>>> b = 'αβψδε\n'.encode()
>>> b
b'\xce\xb1\xce\xb2\xcf\x88\xce\xb4\xce\xb5\n'
>>> for c in b: _ = raw_write(bytes([c]))
...
αβψδε

Here it is on Windows with your patch:

>>> raw_write = sys.stdout.buffer.raw.write
>>> b = 'αβψδε\n'.encode()
>>> b
b'\xce\xb1\xce\xb2\xcf\x88\xce\xb4\xce\xb5\n'
>>> for c in b: _ = raw_write(bytes([c]))
...
��

For the write case this can be addressed by identifying an incomplete
sequence at the tail end and either buffering it as 'written' or
rejecting it for the user/buffer to try again with the complete
sequence. I think rejection isn't a good option when the incomplete
sequence starts at index 0. That should be buffered. I prefer
buffering in all cases.

>> It would probably be simpler to use UTF-16 in the main pipeline and
>> implement Martin's suggestion to mix in a UTF-8 buffer. The UTF-16
>> buffer could be renamed as "wbuffer", for expert use. However, if
>> you're fully committed to transcoding in the raw layer, I'm certain
>> that these problems can be addressed with small buffers and using
>> Python's codec machinery for a flexible mix of "surrogatepass" and
>> "replace" error handling.
>
> I don't think it actually makes things simpler. Having two buffers is
> generally a bad idea unless they are perfectly synced, which would be
> impossible here without data corruption (if you read half a utf-8 character
> sequence and then read the wide buffer, do you get that character or not?).

Martin's idea, as I understand it, is a UTF-8 buffer that reads from
and writes to the text wrapper. It necessarily consumes at least one
character and buffers it to allow reading per byte. Likewise for
writing, it buffers bytes until it can write a character to the text
wrapper. ISTM, it has to look for incomplete lead-continuation byte
sequences at the tail end, to hold them until the sequence is
complete, at which time it either decodes to a valid character or the
U+FFFD replacement character.

Also, I found that read(n) has to read a character at a time. That's
the only way to emulate line-input mode to detect "\n" and stop
reading. Technically this is implemented in a RawIOBase, which
dictates that operations should use a single system call, but since
it's interfacing with a text wrapper around a buffer around the actual
UCS-2 raw console stream, any notion of a 'system call' would be a
sham.

Because of the UTF-8 buffering there is a synchronization issue, but
it has character granularity. For example, when decoding UTF-8, you
don't get half of a surrogate pair. You decode the full character, and
write that as a discrete unit to the text wrapper. I'd have to
experiment to see how bad this can get. If it's too confusing the idea
isn't practical.

On the plus side, when working with text it's all native UCS-2 up to
the TextIOWrapper, so it's as efficient as possible, and as simple as
possible. You don't have to worry about transcoding and dealing with
partial surrogate pairs and partial UTF-8 sequences. All of that
complexity is exported to the pure-Python UTF-8 buffer mixin, but it's
not as bad there either because the interface is Text <=> WTF-8
instead 

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Steve Dower]

On 05Sep2016 1308, Paul Moore wrote:

On 5 September 2016 at 20:30, Steve Dower  wrote:

The only case we can reasonably handle at the raw layer is "n / 4" is zero
but n != 0, in which case we can read and cache up to 4 bytes (one wchar_t)
and then return those in future calls. If we try to cache any more than that
we're substituting for buffered reader, which I don't want to do.

Does caching up to one (Unicode) character at a time sound reasonable? I
think that won't be much trouble, since there's no interference between
system calls in that case and it will be consistent with POSIX behaviour.


Caching a single character sounds perfectly OK. As I noted previously,
my use case probably won't need to work at the raw level anyway, so I
no longer expect to have code that will break, but I think that a
1-character buffer ensuring that we avoid surprises for code that was
written for POSIX is a good trade-off.


So it works, though the behaviour is a little strange when you do it 
from the interactive prompt:


>>> sys.stdin.buffer.raw.read(1)
ɒprint('hi')
b'\xc9'
>>> hi
>>> sys.stdin.buffer.raw.read(1)
b'\x92'
>>>

What happens here is the raw.read(1) rounds one byte up to one 
character, reads the turned alpha, returns a single byte of the two byte 
encoded form and caches the second byte. Then interactive mode reads 
from stdin and gets the rest of the characters, starting from the 
print() and executes that. Finally the next call to raw.read(1) returns 
the cached second byte of the turned alpha.


This is basically only a problem because the readline implementation is 
totally separate from the stdin object and doesn't know about the small 
cache (and for now, I think it's going to stay that way - merging 
readline and stdin would be great, but is a fairly significant task that 
won't make 3.6 at this stage).


I feel like this is an acceptable edge case, as it will only show up 
when interleaving calls to raw.read(n < 4) with multibyte characters and 
input()/interactive prompts. We've taken the 99% compatible to 99.99% 
compatible, and I feel like going any further is practically certain to 
introduce bugs (I'm being very careful with the single character 
buffering, but even that feels risky). Hopefully others agree with my 
risk assessment here, but speak up if you think it's worthwhile trying 
to deal with this final case.


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do PEP 526 type declarations define the types of variables or not?

[Greg Ewing]

Mark Shannon wrote:

Unless of course, others may have a different idea of what the "type of 
a variable" means.

To me, it means it means that for all assignments `var = expr`
the type of `expr` must be a subtype of the variable,
and for all uses of var, the type of the use is the same as the type of 
the variable.


I think it means that, at any given point in time, the
value of the variable is of the type of the variable or
some subtype thereof. That interpretation leaves the
type checker free to make more precise inferences if
it can. For example, in...


def foo()->int:
x:Optional[int] = bar()
if x is None:
return -1
return x


...the type checker could notice that, on the branch
containing 'return x', the value of x must be of type
int, so the code is okay.

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do PEP 526 type declarations define the types of variables or not?

[Peter Ludemann via Python-Dev]

I would take the opposite approach from Greg Ewing, namely that the
annotation is not a permission of values but a starting point for the type
inferencer; and the type checker/inferencer can complain if there's an
inconsistency (for some definition of "inconsistency", which is not defined
in the PEP). In most cases, this distinction doesn't matter, but it does
affect what kinds of errors or warnings are generated.

But ... perhaps people are overthinking these things? If we go back to the
example without variable annotation:

def bar()->Optional[int]: ...

def foo():
  x = bar()
  if x is None:
return -1
  return x

then a straightforward flow-tracing type inferencer can *infer* all the
annotations in foo:

def foo() -> int:  # *not* Optional[int] - see below
  x:Optional[int] = bar()  # derived from definition of bar
  if x is None:  # consistent with x:Optional[int]
return -1  # implies return type of foo
  return x  # implies return type of foo as Union[int, None] minus
None, that is: int

That is, the type annotations add no information in this example, but might
be useful to a human. Perhaps they wouldn't show in the source code at all,
but would instead be put into a database, for use by development tools -
for example, Kythe -flavored
tools, where the type data (and other usage information) are used for code
search, editing, refactoring, etc. (Or the type information could be kept
in a .pyi stub file, with an automated "merge" tool putting them into the
.py file as desired.)

On the other hand, a non-flow-tracing inferencer would derive 'def foo() ->
Optional[int]' ... it would be a *design choice* of the type
checker/inferencer as to whether that's an error, a warning, or silently
allowed ... I can see arguments for all of these choices.

In most cases, there's seldom any need for the programmer to add
annotations to local variables. Global variables and class/instance
attributes, however, can benefit from annotation.

(As to my credentials, which some people seem to crave: I worked on an
earlier version of Google's Python type inferencer (*pytype*) and I'm
currently working on *pykythe *(to be open-sourced), which takes the
function-level information and propagates it to the local variables, then
adds that information (together with call graph information) to a Kythe
database.)



On 5 September 2016 at 15:16, Greg Ewing 
wrote:

> Mark Shannon wrote:
>
> Unless of course, others may have a different idea of what the "type of a
>> variable" means.
>> To me, it means it means that for all assignments `var = expr`
>> the type of `expr` must be a subtype of the variable,
>> and for all uses of var, the type of the use is the same as the type of
>> the variable.
>>
>
> I think it means that, at any given point in time, the
> value of the variable is of the type of the variable or
> some subtype thereof. That interpretation leaves the
> type checker free to make more precise inferences if
> it can. For example, in...
>
> def foo()->int:
>> x:Optional[int] = bar()
>> if x is None:
>> return -1
>> return x
>>
>
> ...the type checker could notice that, on the branch
> containing 'return x', the value of x must be of type
> int, so the code is okay.
>
> --
> Greg
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/pludemann
> %40google.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do PEP 526 type declarations define the types of variables or not?

[Sven R. Kunze]

Didn't Koos say this works more like an expression annotation?

IMO, the type of the expression is what is specified but the type of the 
variable can change over time (as you demonstrated).


Sven


PS: thinking this way, the new syntax is actually confusing as it 
annotates the variable not the expression. :-/



On 05.09.2016 17:26, Mark Shannon wrote:

Hi,

PEP 526 states that "This PEP aims at adding syntax to Python for 
annotating the types of variables" and Guido seems quite insistent 
that the declarations are for the types of variables.


However, I get the impression that most (all) of the authors and 
proponents of PEP 526 are quite keen to emphasise that the PEP in no 
way limits type checkers from doing what they want.


This is rather contradictory. The behaviour of a typechecker is 
defined by the typesystem that it implements. Whether a type 
annotation determines the type of a variable or an expression alters 
changes what typesystems are feasible. So, stating that annotations 
define the type of variables *does* limit what a typechecker can or 
cannot do.


Unless of course, others may have a different idea of what the "type 
of a variable" means.

To me, it means it means that for all assignments `var = expr`
the type of `expr` must be a subtype of the variable,
and for all uses of var, the type of the use is the same as the type 
of the variable.


In this example:

def bar()->Optional[int]: ...

def foo()->int:
x:Optional[int] = bar()
if x is None:
return -1
return x

According to PEP 526 the annotation `x:Optional[int]`
means that the *variable* `x` has the type `Optional[int]`.
So what is the type of `x` in `return x`?
If it is `Optional[int]`, then a type checker is obliged to reject 
this code. If it is `int` then what does "type of a variable" actually 
mean,

and why aren't the other uses of `x` int as well?

Cheers,
Mark.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do PEP 526 type declarations define the types of variables or not?

[Koos Zevenhoven]

On Tue, Sep 6, 2016 at 1:49 AM, Sven R. Kunze  wrote:
> Didn't Koos say this works more like an expression annotation?
>
> IMO, the type of the expression is what is specified but the type of the
> variable can change over time (as you demonstrated).

That's exactly the kind of semantics I'm describing in the
python-ideas thread. An that's exactly how Python works: the type of a
variable can change every time you assign a value to it (but not in
between, unless you're doing funny stuff). So in a sense you annotate
the *value* by annotating the variable at the point in the function
where the value is assigned to it.

There are open questions in this approach of course. But if you're
interested, don't hesitate to discuss or ask questions in the
python-ideas thread. I won't answer before I wake up, though ;).

-- Koos


>
> Sven
>
>
> PS: thinking this way, the new syntax is actually confusing as it annotates
> the variable not the expression. :-/
>
>
>
> On 05.09.2016 17:26, Mark Shannon wrote:
>>
>> Hi,
>>
>> PEP 526 states that "This PEP aims at adding syntax to Python for
>> annotating the types of variables" and Guido seems quite insistent that the
>> declarations are for the types of variables.
>>
>> However, I get the impression that most (all) of the authors and
>> proponents of PEP 526 are quite keen to emphasise that the PEP in no way
>> limits type checkers from doing what they want.
>>
>> This is rather contradictory. The behaviour of a typechecker is defined by
>> the typesystem that it implements. Whether a type annotation determines the
>> type of a variable or an expression alters changes what typesystems are
>> feasible. So, stating that annotations define the type of variables *does*
>> limit what a typechecker can or cannot do.
>>
>> Unless of course, others may have a different idea of what the "type of a
>> variable" means.
>> To me, it means it means that for all assignments `var = expr`
>> the type of `expr` must be a subtype of the variable,
>> and for all uses of var, the type of the use is the same as the type of
>> the variable.
>>
>> In this example:
>>
>> def bar()->Optional[int]: ...
>>
>> def foo()->int:
>> x:Optional[int] = bar()
>> if x is None:
>> return -1
>> return x
>>
>> According to PEP 526 the annotation `x:Optional[int]`
>> means that the *variable* `x` has the type `Optional[int]`.
>> So what is the type of `x` in `return x`?
>> If it is `Optional[int]`, then a type checker is obliged to reject this
>> code. If it is `int` then what does "type of a variable" actually mean,
>> and why aren't the other uses of `x` int as well?
>>
>> Cheers,
>> Mark.
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
>
>
>



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Where are the list and array.array implementations in CPython source?

[Jonathan Goble]

I'd like to study the CPython implementations of lists and array.array
instances for a personal project of mine, but I've very unfamiliar
with the Python source code as it pertains to internals like this.
Which files would I need to look at to do this, and are there a few
particular functions/structures I should pay attention to? I'm just
looking for a brief pointer in the right direction here, not a full
explanation of how it works -- I'll get that from studying the source
code. :-)

(Sorry if this isn't the best place to post, but I felt that a
question about CPython's internals fit slightly better on python-dev
rather than python-list, since this is where those familiar with that
code are more likely to see the post.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Where are the list and array.array implementations in CPython source?

[Martin Panter]

On 5 September 2016 at 23:45, Jonathan Goble  wrote:
> I'd like to study the CPython implementations of lists and array.array
> instances for a personal project of mine, but I've very unfamiliar
> with the Python source code as it pertains to internals like this.
> Which files would I need to look at to do this,

Built-in objects are usually in the Objects/ directory, with a
corresponding include file in the Include/ directory:
https://hg.python.org/cpython/file/default/Objects/listobject.c
https://hg.python.org/cpython/file/default/Include/listobject.h

Modules implemented in C are usually in the Modules/ directory:
https://hg.python.org/cpython/file/default/Modules/arraymodule.c

> and are there a few
> particular functions/structures I should pay attention to? I'm just
> looking for a brief pointer in the right direction here, not a full
> explanation of how it works -- I'll get that from studying the source
> code. :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Where are the list and array.array implementations in CPython source?

[Benjamin Peterson]

Include/listobject.h
Objects/listobject.c
Modules/arraymodule.c

On Mon, Sep 5, 2016, at 16:45, Jonathan Goble wrote:
> I'd like to study the CPython implementations of lists and array.array
> instances for a personal project of mine, but I've very unfamiliar
> with the Python source code as it pertains to internals like this.
> Which files would I need to look at to do this, and are there a few
> particular functions/structures I should pay attention to? I'm just
> looking for a brief pointer in the right direction here, not a full
> explanation of how it works -- I'll get that from studying the source
> code. :-)
> 
> (Sorry if this isn't the best place to post, but I felt that a
> question about CPython's internals fit slightly better on python-dev
> rather than python-list, since this is where those familiar with that
> code are more likely to see the post.)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[eryk sun]

On Mon, Sep 5, 2016 at 9:45 PM, Steve Dower  wrote:
>
> So it works, though the behaviour is a little strange when you do it from
> the interactive prompt:
>
 sys.stdin.buffer.raw.read(1)
> ɒprint('hi')
> b'\xc9'
 hi
 sys.stdin.buffer.raw.read(1)
> b'\x92'

>
> What happens here is the raw.read(1) rounds one byte up to one character,
> reads the turned alpha, returns a single byte of the two byte encoded form
> and caches the second byte. Then interactive mode reads from stdin and gets
> the rest of the characters, starting from the print() and executes that.
> Finally the next call to raw.read(1) returns the cached second byte of the
> turned alpha.
>
> This is basically only a problem because the readline implementation is
> totally separate from the stdin object and doesn't know about the small
> cache (and for now, I think it's going to stay that way - merging readline
> and stdin would be great, but is a fairly significant task that won't make
> 3.6 at this stage).

It needs to read a minimum of 2 codes in case the first character is a
lead surrogate. It can use a length 2 WCHAR buffer and remember how
many bytes have been written (for the general case -- not specifically
for this case).

Example failure using your 3rd patch:

>>> _ = write_console_input("\U0001print('hi')\r\n");\
... raw_read(1)
𐀀print('hi')
b'\xef'
>>>   File "", line 1
�print('hi')
 ^
SyntaxError: invalid character in identifier
>>> raw_read(1)
b'\xbf'
>>> raw_read(1)
b'\xbd'

The raw read captures the first surrogate code, and transcodes it as
the replacement character b'\xef\xbf\xbd' (U+FFFD). Then PyOS_Readline
captures the 2nd surrogate and decodes it as the replacement
character.

In the general case in which a lead surrogate is the last code read,
but not at index 0, it can use the internal buffer to save the code
for the next call.

Surrogates that aren't in valid pairs should be allowed to pass
through via surrogatepass. This aims for consistency with the
filesystem encoding PEP.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Where are the list and array.array implementations in CPython source?

[Jonathan Goble]

On Mon, Sep 5, 2016 at 8:10 PM, Martin Panter  wrote:
> Built-in objects are usually in the Objects/ directory, with a
> corresponding include file in the Include/ directory:
> https://hg.python.org/cpython/file/default/Objects/listobject.c
> https://hg.python.org/cpython/file/default/Include/listobject.h
>
> Modules implemented in C are usually in the Modules/ directory:
> https://hg.python.org/cpython/file/default/Modules/arraymodule.c


On Mon, Sep 5, 2016 at 8:10 PM, Benjamin Peterson  wrote:
> Include/listobject.h
> Objects/listobject.c
> Modules/arraymodule.c


Thanks to both of you. I'll start looking at those soon. :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 6 September 2016 at 04:15, Guido van Rossum  wrote:
> On Mon, Sep 5, 2016 at 10:24 AM, Ethan Furman  wrote:
>> On 09/05/2016 06:46 AM, Nick Coghlan wrote:
>>
>> [an easy to understand explanation for those of us who aren't type-inferring
>> gurus]
>>
>> Thanks, Nick.  I think I finally have a grip on what Mark was talking about,
>> and about how these things should work.
>>
>> Much appreciated!
>
> There must be some misunderstanding. The message from Nick with that
> timestamp 
> (https://mail.python.org/pipermail/python-dev/2016-September/146200.html)
> hinges on an incorrect understanding of the intention of annotations
> without value (e.g. `x: Optional[int]`), leading to a -1 on the PEP.

Short version of below: after sleeping on it, I'd be OK with the PEP
again if it just *added* the explicit type assertions, such that the
shorthand notation could be described in those terms.

Specifically, "x: T = expr" would be syntactic sugar for:

x = expr
assert x: T

While the bare "x: T" would be syntactic sugar for:

assert all(x): T

which in turn would imply that all future bindings of that assignment
target should be accompanied by a type assertion (and typecheckers may
differ in how they define "all future bindings").

Even if everyone always writes the short forms, the explicit
assertions become a useful aid in explaining what those short forms
mean.

The main exploratory question pushed back to the typechecking
community to answer by 3.7 would then be to resolve precisely what
"assert all(TARGET): ANNOTATION" means for different kinds of target
and for different scopes (e.g. constraining nonlocal name rebindings
in closures, constraining attribute rebinding in modules, classes, and
instances).

> I can't tell if this is an honest misunderstanding or a strawman, but
> I want to set the intention straight.

I'm pretty sure I understand your intentions (and broadly agree with
them), I just also agree with Mark that people are going to need some
pretty strong hints that these are not Java/C/C++/C# style type
declarations, and am suggesting a different way of getting there by
being more prescriptive about your intended semantics.

Specifically:

* for 3.6, push everything into a new form of assert statement and
define those assertions as syntactic sugar for PEP 484 constructs
* for 3.7 (and provisionally in 3.6), consider blessing some of those
assertions with the bare annotation syntax

Folks are already comfortable with the notion of assertions not
necessarily being executed at runtime, and they're also comfortable
with them as a way of doing embedded correctness testing inline with
the code.

> First of all, the PEP does not require the type checker to interpret
> anything in a particular way; it intentionally shies away from
> prescribing semantics (other than the runtime semantics of updating
> __annotations__ or verifying that the target appears assignable).

Unfortunately, the ordering problem the PEP introduces means it pushes
very heavily in a particular direction, such that I think we're going
to be better off if you actually specify draft semantics in the PEP
(in terms of existing PEP 484 annotations), rather than leaving it
completely open to interpretation. It's still provisional so you can
change your mind later, but the notion of describing a not yet bound
name is novel enough that I think more guidance (even if it's
provisional) is needed here than was needed in the case of function
annotations.

(I realise you already understand most of the background I go through
below - I'm spelling out my reasoning so you can hopefully figure out
where I'm diverging from your point of view)

If we look at PEP 484, all uses of annotations exist between two
pieces of code: one that produces a value, and one that binds the
value to a reference.

As such, they act as type assertions:

- on parameters, they assert "I am expecting this argument to be of this type"
- on assignments, they assert "I an expecting this initialiser to be
of this type"

Typecheckers can then use those assertions in two ways: as a
constraint on the value producer, and as a more precise hint if type
inference either isn't possible (e.g. function parameters,
initialisation to None), or gives an overly broad answer (e.g empty
containers)

The "x: T = expr" syntax is entirely conformant with that system - all
it does is change the spelling of the existing type hint comments.

Allowing "assert x: T" would permit that existing kind of type
assertion to be inserted at arbitrary points in the code without
otherwise affecting control flow or type inference, as if you had
written:

 # PEP 484
 def is_T(arg: T) -> None:
 pass

is_T(x)

Or:

# PEP 526
x: T = x

By contrast, bare annotations on new assignment targets without an
initialiser can't be interpreted that way, as there is no *preceding
value to constrain*.

That inability to interpret them in the same sense as existing
annotations means that ther

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

I'm sorry, but we're not going to invent new syntax this late in the
game. The syntax proposed by the PEP has been on my mind ever since
PEP 484 with very minor variations; I first proposed it seriously on
python-ideas over a month ago, we've been debating the details since
then, and it's got a solid implementation based on those debates by
Ivan Levkivskyi. In contrast, it looks like you just made the "assert
x: T" syntax up last night in response to the worries expressed by
Mark Shannon, and "assert" sounds a lot like a run-time constraint to
me.

Instead, I encourage you to participate in the writing of a separate
PEP explaining how type checkers are expected to work (since PEP 526
doesn't specify that). Ivan is also interested in such a PEP and we
hope Mark will also lend us his expertise.

On Mon, Sep 5, 2016 at 8:46 PM, Nick Coghlan  wrote:
> On 6 September 2016 at 04:15, Guido van Rossum  wrote:
>> On Mon, Sep 5, 2016 at 10:24 AM, Ethan Furman  wrote:
>>> On 09/05/2016 06:46 AM, Nick Coghlan wrote:
>>>
>>> [an easy to understand explanation for those of us who aren't type-inferring
>>> gurus]
>>>
>>> Thanks, Nick.  I think I finally have a grip on what Mark was talking about,
>>> and about how these things should work.
>>>
>>> Much appreciated!
>>
>> There must be some misunderstanding. The message from Nick with that
>> timestamp 
>> (https://mail.python.org/pipermail/python-dev/2016-September/146200.html)
>> hinges on an incorrect understanding of the intention of annotations
>> without value (e.g. `x: Optional[int]`), leading to a -1 on the PEP.
>
> Short version of below: after sleeping on it, I'd be OK with the PEP
> again if it just *added* the explicit type assertions, such that the
> shorthand notation could be described in those terms.
>
> Specifically, "x: T = expr" would be syntactic sugar for:
>
> x = expr
> assert x: T
>
> While the bare "x: T" would be syntactic sugar for:
>
> assert all(x): T
>
> which in turn would imply that all future bindings of that assignment
> target should be accompanied by a type assertion (and typecheckers may
> differ in how they define "all future bindings").
>
> Even if everyone always writes the short forms, the explicit
> assertions become a useful aid in explaining what those short forms
> mean.
>
> The main exploratory question pushed back to the typechecking
> community to answer by 3.7 would then be to resolve precisely what
> "assert all(TARGET): ANNOTATION" means for different kinds of target
> and for different scopes (e.g. constraining nonlocal name rebindings
> in closures, constraining attribute rebinding in modules, classes, and
> instances).
>
>> I can't tell if this is an honest misunderstanding or a strawman, but
>> I want to set the intention straight.
>
> I'm pretty sure I understand your intentions (and broadly agree with
> them), I just also agree with Mark that people are going to need some
> pretty strong hints that these are not Java/C/C++/C# style type
> declarations, and am suggesting a different way of getting there by
> being more prescriptive about your intended semantics.
>
> Specifically:
>
> * for 3.6, push everything into a new form of assert statement and
> define those assertions as syntactic sugar for PEP 484 constructs
> * for 3.7 (and provisionally in 3.6), consider blessing some of those
> assertions with the bare annotation syntax
>
> Folks are already comfortable with the notion of assertions not
> necessarily being executed at runtime, and they're also comfortable
> with them as a way of doing embedded correctness testing inline with
> the code.
>
>> First of all, the PEP does not require the type checker to interpret
>> anything in a particular way; it intentionally shies away from
>> prescribing semantics (other than the runtime semantics of updating
>> __annotations__ or verifying that the target appears assignable).
>
> Unfortunately, the ordering problem the PEP introduces means it pushes
> very heavily in a particular direction, such that I think we're going
> to be better off if you actually specify draft semantics in the PEP
> (in terms of existing PEP 484 annotations), rather than leaving it
> completely open to interpretation. It's still provisional so you can
> change your mind later, but the notion of describing a not yet bound
> name is novel enough that I think more guidance (even if it's
> provisional) is needed here than was needed in the case of function
> annotations.
>
> (I realise you already understand most of the background I go through
> below - I'm spelling out my reasoning so you can hopefully figure out
> where I'm diverging from your point of view)
>
> If we look at PEP 484, all uses of annotations exist between two
> pieces of code: one that produces a value, and one that binds the
> value to a reference.
>
> As such, they act as type assertions:
>
> - on parameters, they assert "I am expecting this argument to be of this type"
> - on assignments, they assert "I an expecting t