Re: [Python-Dev] pathlib - current status of discussions

[Nick Coghlan]

On 15 April 2016 at 00:01, Random832  wrote:
> On Thu, Apr 14, 2016, at 09:50, Chris Angelico wrote:
>> Adding integers and floats is considered "safe" because most people's
>> use of floats completely compasses their use of ints. (You'll get
>> OverflowError if it can't be represented.) But float and Decimal are
>> considered "unsafe":
>>
>> >>> 1.5 + decimal.Decimal("1.5")
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> TypeError: unsupported operand type(s) for +: 'float' and
>> 'decimal.Decimal'
>>
>> This is more what's happening here. Floats and Decimals can represent
>> similar sorts of things, but with enough incompatibilities that you
>> can't simply merge them.
>
> And what such incompatibilities exist between bytes and str for the
> purpose of representing file paths? At the end of the day, there's
> exactly one answer to "what file on disk this represents (or would
> represent if it existed)".

Bytes paths on WIndows are encoded as mbcs for use with the ASCII-only
Windows APIs, and hence don't support the full range of characters
that str does. The colloquial shorthand for that is "bytes paths don't
work properly on Windows" (the more strictly accurate description is
"bytes paths only work correctly on Windows if every code point in the
path can be encoded using the 'mbcs' codec").

Even on *nix, os.fsencode may fail outright if the system is
configured to use a non-universal encoding, while os.fsdecode may
pollute the resulting string with surrogate escaped characters.

Regardless of platform, if somebody hands you *mixed* bytes and str
data, the appropriate default reaction is to complain about it rather
than assume they meant one or the other. That complaint may take one
of two forms:

- for a high level, platform independent API, bytes should just be
rejected outright
- for a low level API with input type dependent behaviour, the input
should be rejected as ambiguous - the API doesn't know whether the str
behaviour or the bytes behaviour is the intended one

pathlib falls into the first category - it just rejects bytes as input
os.path.join falls into the second category - all str is fine, and all
bytes is fine, but mixing them fails

However, once somebody reaches for the coercion APIs (fsdecode and
fsencode), they're now *explicitly* telling the interpreter what they
want, since there's no ambiguity about the possible return types from
those functions.

In relation to Victor's comment about this being complex code to show
to a novice:

  os.path.join(*map(os.fsdecode, ("str", b"bytes")))

I agree, but also think that's a good reason for people to switch to
teaching novices pathlib rather than os.path, and letting them
discover the underlying libraries as required by the code and examples
they encounter.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Victor Stinner]

Le vendredi 15 avril 2016, Stefan Behnel  a écrit :

> How can that be achieved? If the tag is just a sequentially growing number,
> creating two dicts and applying one operation to the first one should give
> both the same version tag, right?
>

Armin didn't propose to get ride of the global version.

a = dict() # version = 0
b = dict() # version = 0
a['key'] = 'value' # version = 300
b['key'] = 'value' # version = 301

Victor
PS: It looks like the iPad Gmail app foces me to use HTML, I don't know how
to use plain text :-/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should secrets include a fallback for hmac.compare_digest?

[Victor Stinner]

It's easy to implement this function (in the native language of your Python
implemenation), it's short. I'm not sure that a Python version is really
safe.

The secrets module is for Python 3.6, in this version the hmac already
"requires" the compare_digest() function no?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Antoine Pitrou]

Victor Stinner  gmail.com> writes:
> 
> Hi,
> 
> 2016-04-14 22:42 GMT+02:00 Armin Rigo  tunes.org>:
> > Hi Victor,
> >
> > On 14 April 2016 at 17:19, Victor Stinner 
gmail.com> wrote:
> >> Each time a dictionary is created, the global
> >> version is incremented and the dictionary version is initialized to the
> >> global version.
> >
> > A detail, but why not set the version tag of new empty dictionaries to
> > zero, always?   Same after a clear().  This would satisfy the
> > condition: equality of the version tag is supposed to mean "the
> > dictionary content is precisely the same".
> 
> You're right that incrementing the global version is useless for these
> specific cases, and using the version 0 should work. It only matters
> that the version (version? version tag?) is different.

Why do this? It's a nice property that two dicts always have different
version tags, and now you're killing this property for... no obvious
reason?

Do you really think dict.clear() is in need of micro-optimizing a
couple CPU cycles away?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Stefan Behnel]

Victor Stinner schrieb am 15.04.2016 um 10:20:
> Le vendredi 15 avril 2016, Stefan Behnel a écrit :
> 
>> How can that be achieved? If the tag is just a sequentially growing number,
>> creating two dicts and applying one operation to the first one should give
>> both the same version tag, right?
>>
> 
> Armin didn't propose to get ride of the global version.
> 
> a = dict() # version = 0
> b = dict() # version = 0
> a['key'] = 'value' # version = 300
> b['key'] = 'value' # version = 301

Ah, sorry, should have read the PEP more closely. It's *always* the global
version that gets incremented. Then yes, that's a safe point of distinction
for dicts and their status.

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should secrets include a fallback for hmac.compare_digest?

[Steven D'Aprano]

On Fri, Apr 15, 2016 at 10:26:31AM +0200, Victor Stinner wrote:
> It's easy to implement this function (in the native language of your Python
> implemenation), it's short. I'm not sure that a Python version is really
> safe.
> 
> The secrets module is for Python 3.6, in this version the hmac already
> "requires" the compare_digest() function no?

The current version looks like this:

try:
   from hmac import compare_digest
except ImportError:
   # fallback version defined


but I'm having second thoughts about this. I don't think it needs to 
support older versions of Python, but perhaps it needs to support 
implementations which don't include compare_digest?

This isn't just a question about the secrets module. PEP 399 suggests 
than any C classes/functions should have a pure Python version as 
fallback, but compare_digest doesn't. I don't know whether it should or 
not.

https://www.python.org/dev/peps/pep-0399/



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Victor Stinner]

2016-04-15 11:01 GMT+02:00 Antoine Pitrou :
> Victor Stinner  gmail.com> writes:
>> You're right that incrementing the global version is useless for these
>> specific cases, and using the version 0 should work. It only matters
>> that the version (version? version tag?) is different.
>
> Why do this? It's a nice property that two dicts always have different
> version tags, and now you're killing this property for... no obvious
> reason?

I guess that the reason is to reduce *a little bit* the risk of
integer overflow (especially the bug when a guard doesn't see a change
between new_version = old_version % 2**64).

> Do you really think dict.clear() is in need of micro-optimizing a
> couple CPU cycles away?

The advantage of having a different version for empty dict is to be
able to use the version to check that they are different. Using the
dictionary pointer is not enough, since it's common that a new
dictionary gets the address of a previously destroyed dictionary. This
case can be avoided if you keep dictionaries alive by keeping a strong
reference, but there are good reasons to not keep a strong reference.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 506 secrets module

[Victor Stinner]

Hi,

Would it make sense to add a function to generate a random UUID4 (as a
string) in secrets?

The current implement in uuid.py of CPython 3.6 already uses os.urandom():

def uuid4():
"""Generate a random UUID."""
return UUID(bytes=os.urandom(16), version=4)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should secrets include a fallback for hmac.compare_digest?

[Victor Stinner]

2016-04-15 11:21 GMT+02:00 Steven D'Aprano :
> This isn't just a question about the secrets module. PEP 399 suggests
> than any C classes/functions should have a pure Python version as
> fallback, but compare_digest doesn't. I don't know whether it should or
> not.

The hmac module is responsible to providing a fallback, not the secrets module.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should secrets include a fallback for hmac.compare_digest?

[Paul Moore]

On 15 April 2016 at 10:35, Victor Stinner  wrote:
> 2016-04-15 11:21 GMT+02:00 Steven D'Aprano :
>> This isn't just a question about the secrets module. PEP 399 suggests
>> than any C classes/functions should have a pure Python version as
>> fallback, but compare_digest doesn't. I don't know whether it should or
>> not.
>
> The hmac module is responsible to providing a fallback, not the secrets 
> module.

Agreed. The library docs state that the hmac module provides
compare_digest, so you are therefore entitled to unconditionally
import it (just as end user code would).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Nick Coghlan]

On 15 April 2016 at 00:52, Stephen J. Turnbull  wrote:
> Nick Coghlan writes:
>
>  > The use case for returning bytes from __fspath__ is DirEntry, so you
>  > can write things like this in low level code:
>  >
>  > def myscandir(dirpath):
>  > for entry in os.scandir(dirpath):
>  > if entry.is_file():
>  > with open(entry) as f:
>  > # do something
>
> Excuse me, but that is *not* a use case for returning bytes from
> DirEntry.__fspath__.  open() is perfectly happy taking str (including
> surrogate-encoded rawbytes).

That results in a different type for the file object's name:

>>> open("README.md").name
'README.md'
>>> open(b"README.md").name
b'README.md'

Implicitly level shifting in a low level API isn't a good thing,
especially when there are idempotent level shifting commands available
(so you can always ensure a given value is on the level you expect,
even if you don't know which level it was on originally).

I completely agree with you that folks working with text in the binary
domain are asking for trouble, but at the same time, that's the
reality of the way a lot of *nix system interfaces operate. The
guarantee we want to provide those folks is that if they're operating
in the binary domain they'll stay there unless they explicitly shift
out of it using a decoding API of some kind - doing it behind their
back would be akin to implicitly shifting from the time domain to the
frequency domain in an engineering library.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 506 secrets module

[Nick Coghlan]

On 15 April 2016 at 19:39, Victor Stinner  wrote:
> Hi,
>
> Would it make sense to add a function to generate a random UUID4 (as a
> string) in secrets?
>
> The current implement in uuid.py of CPython 3.6 already uses os.urandom():
>
> def uuid4():
> """Generate a random UUID."""
> return UUID(bytes=os.urandom(16), version=4)

I don't think so, as folks looking to generate a UUID specifically are
already likely to end up at the uuid module docs rather than trying to
craft their own based on the random module (and the uuid module
already does the right thing, and it would be a bug if it didn't).

The new secrets module fills the gap for cases where random is
otherwise an attractive nuisance by making it easy to say "use this
instead".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should secrets include a fallback for hmac.compare_digest?

[Nick Coghlan]

On 15 April 2016 at 14:52, Steven D'Aprano  wrote:
> Now that PEP 506 has been approved, I've checked in the secrets module,
> but an implementation question has come up regarding compare_digest.
>
> Currently, the module tries to import hmac.compare_digest, and if that
> fails, then it falls back to a Python version. But since compare_digest
> has been available since 3.3, I'm now questioning whether the fallback
> is useful at all. Perhaps for alternate Python implementations?
>
> So, two questions:
>
> - should secrets include a fallback?

It definitely *shouldn't* include a fallback, as the function needs to
be writen in C (or some other not-normal-Python-code language) in
order to provide the appropriate timing guarantees.

We added hmac.compare_digest in response to Python web frameworks
providing their own pure Python "constant time" comparison functions
that were nevertheless still subject to remote timing atacks.

I'd forgotten about the hmac vs operator indirection, but it's still
better to import the public API from hmac (since
operator._compare_digest is a Python implementation detail, and you
may as well make it easy to extract the secrets module for use in
earlier versions - 2.7 also gained hmac.compare_digest as part of PEP
466).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2016-04-08 - 2016-04-15)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open5489 (+12)
  closed 33039 (+46)
  total  38528 (+58)

Open issues with patches: 2381 


Issues opened (45)
==

#11205: Evaluation order of dictionary display is different from refer
http://bugs.python.org/issue11205  reopened by ncoghlan

#25609: Add a ContextManager ABC and type
http://bugs.python.org/issue25609  reopened by brett.cannon

#25731: Assigning and deleting __new__ attr on the class does not allo
http://bugs.python.org/issue25731  reopened by barry

#26673: Tkinter error when opening IDLE configuration menu
http://bugs.python.org/issue26673  reopened by terry.reedy

#26716: EINTR handling in fcntl
http://bugs.python.org/issue26716  opened by Jack Zhou

#26717: wsgiref.simple_server: mojibake with cp1252 bytes in PATH_INFO
http://bugs.python.org/issue26717  opened by Anthony Sottile

#26720: memoryview from BufferedWriter becomes garbage
http://bugs.python.org/issue26720  opened by martin.panter

#26721: Avoid socketserver.StreamRequestHandler.wfile doing partial wr
http://bugs.python.org/issue26721  opened by martin.panter

#26724: Serialize dict with non-string keys to JSON — unexpected res
http://bugs.python.org/issue26724  opened by anton-ryzhov

#26726: Incomplete Internationalization in Argparse Module
http://bugs.python.org/issue26726  opened by IronGrid

#26728: make pdb.set_trace() accept debugger commands as arguments and
http://bugs.python.org/issue26728  opened by irdb

#26729: Incorrect __text_signature__ for sorted
http://bugs.python.org/issue26729  opened by eriknw

#26730: SpooledTemporaryFile doesn't correctly preserve data for text 
http://bugs.python.org/issue26730  opened by James Hennessy

#26731: subprocess on windows leaks stdout/stderr handle to child proc
http://bugs.python.org/issue26731  opened by saifujinaro

#26732: multiprocessing sentinel resource leak
http://bugs.python.org/issue26732  opened by quick-b

#26733: staticmethod and classmethod are ignored when disassemble clas
http://bugs.python.org/issue26733  opened by xiang.zhang

#26736: Use HTTPS protocol in links
http://bugs.python.org/issue26736  opened by serhiy.storchaka

#26739: idle: Errno 10035 a non-blocking socket operation could not be
http://bugs.python.org/issue26739  opened by MICHAEL JACOBSON

#26740: tarfile: accessing (listing and extracting) tarball fails with
http://bugs.python.org/issue26740  opened by Tomas Tomecek

#26741: subprocess.Popen should emit a ResourceWarning in destructor i
http://bugs.python.org/issue26741  opened by haypo

#26742: imports in test_warnings changes warnings.filters
http://bugs.python.org/issue26742  opened by haypo

#26743: Unable to import random with python2.7 on power pc based machi
http://bugs.python.org/issue26743  opened by ragreddy

#26744: print() function hangs on MS-Windows 10
http://bugs.python.org/issue26744  opened by Ma Lin

#26745: Redundant code in _PyObject_GenericSetAttrWithDict
http://bugs.python.org/issue26745  opened by xiang.zhang

#26746: struct.pack(): trailing padding bytes on x64
http://bugs.python.org/issue26746  opened by skrah

#26750: Mock autospec does not work with subclasses of property()
http://bugs.python.org/issue26750  opened by amaury.forgeotdarc

#26751: Possible bug in sorting algorithm
http://bugs.python.org/issue26751  opened by David.Manowitz

#26752: Mock(2.0.0).assert_has_calls() raise AssertionError in two sam
http://bugs.python.org/issue26752  opened by jekin000

#26753: Obmalloc lock LOCK_INIT and LOCK_FINI are never used
http://bugs.python.org/issue26753  opened by larry

#26754: PyUnicode_FSDecoder() accepts arbitrary iterable
http://bugs.python.org/issue26754  opened by serhiy.storchaka

#26755: Update version{added,changed} docs in devguide
http://bugs.python.org/issue26755  opened by berker.peksag

#26756: fileinput handling of unicode errors from standard input
http://bugs.python.org/issue26756  opened by jmb236

#26757: test_urllib2net.test_http_basic() timeout after 15 min on
http://bugs.python.org/issue26757  opened by haypo

#26758: Unnecessary format string handling for no argument slot wrappe
http://bugs.python.org/issue26758  opened by josh.r

#26759: PyBytes_FromObject accepts arbitrary iterable
http://bugs.python.org/issue26759  opened by serhiy.storchaka

#26760: Document PyFrameObject
http://bugs.python.org/issue26760  opened by brett.cannon

#26762: test_multiprocessing_spawn leaves processes running in backgro
http://bugs.python.org/issue26762  opened by martin.panter

#26763: Update PEP-8 regarding binary operators
http://bugs.python.org/issue26763  opened by IanLee1521

#26764: SystemError in bytes.__rmod__
http://bugs.python.org/issue26764  opened by serhiy.storchaka

#26765: Factor out common bytes and bytearray implementation
http://bugs.python.org/

[Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Guido van Rossum]

After a fruitful discussion on python-ideas I've decided that it's fine to
break lines *before* a binary operator. It looks better and Knuth
recommends it.

The head of the python-ideas discussion:
https://mail.python.org/pipermail/python-ideas/2016-April/039752.html

See also the discussion in the tracker: http://bugs.python.org/issue26763

Here's the diff I applied: https://hg.python.org/peps/rev/3857909d7956

The talk by Brandon Rhodes where Knuth is referenced ([3] below):
http://rhodesmill.org/brandon/slides/2012-11-pyconca/#laying-down-the-law

The key section in PEP 8 that was updated (apart from fixing up references):

Should a line break before or after a binary operator?
--

For decades the recommended style has been to break after binary
operators.  However, recent reseach unearthed recommendations by
Donald Knuth to break *before* binary operators, in his writings about
typesetting [3]_.  Therefore it is permissible to break before or
after a binary operator, as long as the convention is consistent
locally.  For new code Knuth's style is suggested.

Some examples of code breaking before binary Boolean operators::

class Rectangle(Blob):

def __init__(self, width, height,
 color='black', emphasis=None, highlight=0):
if (width == 0
and height == 0
and color == 'red'
and emphasis == 'strong'
or highlight > 100):
raise ValueError("sorry, you lose")
if (width == 0 and height == 0
and (color == 'red' or emphasis is None)):
raise ValueError("I don't think so -- values are %s, %s" %
 (width, height))
Blob.__init__(self, width, height,
  color, emphasis, highlight)


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Victor Stinner]

Hum.

if (width == 0
and height == 0
and color == 'red'
and emphasis == 'strong'
or highlight > 100):
raise ValueError("sorry, you lose")

Please remove one space to vertically align "and" operators with the
opening parenthesis:

if (width == 0
   and height == 0
   and color == 'red'
   and emphasis == 'strong'
   or highlight > 100):
raise ValueError("sorry, you lose")

(I'm not sure that the difference is obvious in a mail client, you
need a fixed width font which is not the case in my Gmail editor.)

It helps to visually see that the multiline test and the raise
instruction are in two different blocks.

(Moreover, the pep8 checks of OpenStack simply reject such syntax, but
I cannot use this syntax anymore :-))

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Tim Peters]

[Guido]
> After a fruitful discussion on python-ideas I've decided that it's fine to
> break lines *before* a binary operator. It looks better and Knuth recommends
> it.
> ...
> Therefore it is permissible to break before or
> after a binary operator, as long as the convention is consistent
> locally.  For new code Knuth's style is suggested.
>
> Some examples of code breaking before binary Boolean operators::
>
> class Rectangle(Blob):
>
> def __init__(self, width, height,
>  color='black', emphasis=None, highlight=0):
> if (width == 0
> and height == 0
> and color == 'red'
> and emphasis == 'strong'
> or highlight > 100):
> raise ValueError("sorry, you lose")
> if (width == 0 and height == 0
> and (color == 'red' or emphasis is None)):
> raise ValueError("I don't think so -- values are %s, %s" %
>  (width, height))
> Blob.__init__(self, width, height,
>   color, emphasis, highlight)
>

Note that this code still breaks a line after a binary operator (the
string formatting "%" operator in the 2nd ValueError call).  But it's
perfectly clear the way it is.  Good taste can't be reduced to rules
;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Gustavo Carneiro]

On 15 April 2016 at 18:03, Victor Stinner  wrote:

> Hum.
>
> if (width == 0
> and height == 0
> and color == 'red'
> and emphasis == 'strong'
> or highlight > 100):
> raise ValueError("sorry, you lose")
>
> Please remove one space to vertically align "and" operators with the
> opening parenthesis:
>
> if (width == 0
>and height == 0
>and color == 'red'
>and emphasis == 'strong'
>or highlight > 100):
> raise ValueError("sorry, you lose")
>

Personally, I think what you propose looks ugly.  The first version looks
so much better.


It helps to visually see that the multiline test and the raise
> instruction are in two different blocks.


The only thing I would add would be an empty line to help distinguish the
if expression block from the "then" code block:

if (width == 0
and height == 0
and color == 'red'
and emphasis == 'strong'
or highlight > 100):

raise ValueError("sorry, you lose")


-- 
Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Guido van Rossum]

On Fri, Apr 15, 2016 at 10:03 AM, Victor Stinner 
wrote:

> Hum.
>
> if (width == 0
> and height == 0
> and color == 'red'
> and emphasis == 'strong'
> or highlight > 100):
> raise ValueError("sorry, you lose")
>
> Please remove one space to vertically align "and" operators with the
> opening parenthesis:
>
> if (width == 0
>and height == 0
>and color == 'red'
>and emphasis == 'strong'
>or highlight > 100):
> raise ValueError("sorry, you lose")
>
> (I'm not sure that the difference is obvious in a mail client, you
> need a fixed width font which is not the case in my Gmail editor.)
>

I can see it perfectly fin and I disagree.


> It helps to visually see that the multiline test and the raise
> instruction are in two different blocks.
>
> (Moreover, the pep8 checks of OpenStack simply reject such syntax, but
> I cannot use this syntax anymore :-))


That's why that tool shouldn't be named after the PEP. See
https://github.com/PyCQA/pycodestyle/issues/466

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Serhiy Storchaka]

On 15.04.16 20:03, Victor Stinner wrote:

Hum.

 if (width == 0
 and height == 0
 and color == 'red'
 and emphasis == 'strong'
 or highlight > 100):
 raise ValueError("sorry, you lose")

Please remove one space to vertically align "and" operators with the
opening parenthesis:

 if (width == 0
and height == 0
and color == 'red'
and emphasis == 'strong'
or highlight > 100):
 raise ValueError("sorry, you lose")


I would rather *add* spaces to wrapped condition lines.

if (width == 0
and height == 0
and color == 'red'
and emphasis == 'strong'
or highlight > 100):
raise ValueError("sorry, you lose")


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Guido van Rossum]

The update is already serving its real purpose: showing that style is
debatable and cannot always easily be reduced to fixed rules.

On Fri, Apr 15, 2016 at 10:24 AM, Serhiy Storchaka 
wrote:

> On 15.04.16 20:03, Victor Stinner wrote:
>
>> Hum.
>>
>>  if (width == 0
>>  and height == 0
>>  and color == 'red'
>>  and emphasis == 'strong'
>>  or highlight > 100):
>>  raise ValueError("sorry, you lose")
>>
>> Please remove one space to vertically align "and" operators with the
>> opening parenthesis:
>>
>>  if (width == 0
>> and height == 0
>> and color == 'red'
>> and emphasis == 'strong'
>> or highlight > 100):
>>  raise ValueError("sorry, you lose")
>>
>
> I would rather *add* spaces to wrapped condition lines.
>
> if (width == 0
> and height == 0
> and color == 'red'
> and emphasis == 'strong'
> or highlight > 100):
> raise ValueError("sorry, you lose")
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[MRAB]

On 2016-04-15 18:03, Victor Stinner wrote:
> Hum.
>
>  if (width == 0
>  and height == 0
>  and color == 'red'
>  and emphasis == 'strong'
>  or highlight > 100):
>  raise ValueError("sorry, you lose")
>
> Please remove one space to vertically align "and" operators with the
> opening parenthesis:
>
>  if (width == 0
> and height == 0
> and color == 'red'
> and emphasis == 'strong'
> or highlight > 100):
>  raise ValueError("sorry, you lose")
>
> (I'm not sure that the difference is obvious in a mail client, you
> need a fixed width font which is not the case in my Gmail editor.)
>
> It helps to visually see that the multiline test and the raise
> instruction are in two different blocks.
>
> (Moreover, the pep8 checks of OpenStack simply reject such syntax, but
> I cannot use this syntax anymore :-))
>
I always half-indent continuation lines:

if (width == 0
  and height == 0
  and color == 'red'
  and emphasis == 'strong'
  or highlight > 100):
raise ValueError("sorry, you lose")
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Jim J. Jewett]

On Thu Apr 14 11:19:42 EDT 2016, Victor Stinner posted the latest
draft of PEP 509; dict version_tag

(1)  Meta Question:  If this is really only for CPython, then is
"Standards Track" the right classification?

(2)  Why *promise* not to update the version_tag when replacing a
value with itself?  Isn't that the sort of quality-of-implementation
issue that got pushed to a note for objects that happen to be
represented as singletons, such as small integers or ASCII chars?

I think it is a helpful optimization, and worth documenting ... I
just think it should be at the layer of "this particular patch",
rather than something that sounds like part of the contract.

e.g.,

... The global version is also incremented and copied to the
dictionary version at each dictionary change.  The following
dict methods can trigger changes:

* ``clear()`` 
* ``pop(key)``
* ``popitem()`` 
* ``setdefault(key, value)`` 
* ``__detitem__(key)`` 
* ``__setitem__(key, value)`` 
* ``update(...)``

.. note::  As a quality of implementation issue, the actual patch
does not increment the version_tag when it can prove that there
was no actual change.  For example, clear() on an already-empty
dict will not trigger a version_tag change, nor will updating a
dict with itself, since the values will be unchanged.  For efficiency,
the analysis considers only object identity (not equality) when
deciding whether to increment the version_tag.

[2A] Do you want to promise that replacing a value with a
non-identical object *will* trigger a version_tag update *even*
if the objects are equal?

I would vote no, but I realize backwards-compatibility may create
such a promise implicitly.

(3)  It is worth being explicit on whether empty dicts can share
a version_tag of 0.  If this PEP is about dict content, then that
seems fine, and it may well be worth optimizing dict creation.

There are times when it is important to keep the same empty dict;
I can't think of any use cases where it is important to verify
that some *other* code has done so, *and* I can't get a reference
to the correct dict for an identity check.

(4)  Please be explicit about the locking around version++; it
is enough to say that the relevant methods already need to hold
the GIL (assuming that is true).

(5)  I'm not sure I understand the arguments around a per-entry
version.

On the one hand, you never need a strong reference to the value;
if it has been collected, then it has obviously been removed from
the dict and should trigger a change even with per-dict.

On the other hand, I'm not sure per-entry would really allow
finer-grained guards to avoid lookups; just because an entry hasn't
been modified doesn't prove it hasn't been moved to another location,
perhaps by replacing a dummy in a slot it would have preferred.

(6)  I'm also not sure why version_tag *doesn't* solve the problem
of dicts that fool the iteration guards by mutating without changing
size ( https://bugs.python.org/issue19332 ) ... are you just saying
that the iterator views aren't allowed to rely on the version-tag
remaining stable, because replacing a value (as opposed to a
key-value pair) is allowed?

I had always viewed the failing iterators as a supporting-this-case-
makes-the-code-too-slow-and-ugly limitation, rather than a data
integrity check.  When I do care about the data not changing,
(an exposed variant of) version_tag is as likely to be what I want as
a hypothetical keys_version_tag would be. 

-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Barry Warsaw]

On Apr 15, 2016, at 09:53 AM, Guido van Rossum wrote:

>After a fruitful discussion on python-ideas I've decided that it's fine to
>break lines *before* a binary operator.

Thanks Guido, your changes look great.

-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Oscar Benjamin]

On 15 April 2016 at 18:54, Jim J. Jewett  wrote:
>
> [2A] Do you want to promise that replacing a value with a
> non-identical object *will* trigger a version_tag update *even*
> if the objects are equal?
>
> I would vote no, but I realize backwards-compatibility may create
> such a promise implicitly.

It needs to trigger a version update. Equality  doesn't guarantee any
kind of equivalence in Python. It's not even guaranteed that a==b will
come to the same value if evaluated twice in a row.

An example:

>>> from fractions import Fraction as F
>>> F(1) == 1
True
>>> d = globals()
>>> d['a'] = F(1)
>>> a.limit_denominator()
Fraction(1, 1)
>>> d['a'] = 1
>>> a.limit_denominator()
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'int' object has no attribute 'limit_denominator'

--
Oscar
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Victor Stinner]

2016-04-15 19:54 GMT+02:00 Jim J. Jewett :
> (1)  Meta Question:  If this is really only for CPython, then is
> "Standards Track" the right classification?

Yes, I think so. It doesn't seem to be an Informal nor a Process:
https://www.python.org/dev/peps/pep-0001/#pep-types


> (2)  Why *promise* not to update the version_tag when replacing a
> value with itself?

It's an useful property. For example, let's say that you have a guard
on globals()['value']. The guard is created with value=3. An unit test
replaces the value with 50, but then restore the value to its previous
value (3). Later, the guard is checked to decide if an optimization
can be used.

If the dict version is increased, you need a lookup. If the dict
version is not increased, the guard is cheap.

In C, it's very cheap to implement the test "new_value == old_value",
it just compares two pointers.

If an overhead is visible, I can drop it from the PEP, and implement
the check in the guard.


>  Isn't that the sort of quality-of-implementation
> issue that got pushed to a note for objects that happen to be
> represented as singletons, such as small integers or ASCII chars?

I prefer to require this property.


> [2A] Do you want to promise that replacing a value with a
> non-identical object *will* trigger a version_tag update *even*
> if the objects are equal?

It's already written in the PEP:

"The version is not incremented if an existing key is set to the same
value. For efficiency, values are compared by their identity:
new_value is old_value , not by their content: new_value == old_value
."


> (3)  It is worth being explicit on whether empty dicts can share
> a version_tag of 0.  If this PEP is about dict content, then that
> seems fine, and it may well be worth optimizing dict creation.

This is not part of the PEP yet. I'm not sure that I will modify the
PEP to use the version 0 for empty dictionaries. Antoine doesn't seem
to be convinced :-)


> (4)  Please be explicit about the locking around version++; it
> is enough to say that the relevant methods already need to hold
> the GIL (assuming that is true).

I don't think that it's important to mention it in the PEP. It's more
an implementation detail. The version can be protected by atomic
operations.


> (5)  I'm not sure I understand the arguments around a per-entry
> version.

It doesn't matter since I don't want this option :-)


> On the one hand, you never need a strong reference to the value;
> if it has been collected, then it has obviously been removed from
> the dict and should trigger a change even with per-dict.

Let's say that you watch the key1 of a dict. The key2 is modified, it
increases the version. Later, you test the guard: to check if the key1
was modified, you need to lookup the key and compare the value. You
need the value to compare it.

> On the other hand, I'm not sure per-entry would really allow
> finer-grained guards to avoid lookups; just because an entry hasn't
> been modified doesn't prove it hasn't been moved to another location,
> perhaps by replacing a dummy in a slot it would have preferred.

The main advantage of per-entry version is to avoid the strong
reference to values.

According to my tests, the drawbacks are too important to take this
option. I prefer a simple version per dictionary.


> (6)  I'm also not sure why version_tag *doesn't* solve the problem
> of dicts that fool the iteration guards by mutating without changing
> size ( https://bugs.python.org/issue19332 ) ... are you just saying
> that the iterator views aren't allowed to rely on the version-tag
> remaining stable, because replacing a value (as opposed to a
> key-value pair) is allowed?

If the dictionary values are modified during the loop, the dict
version is increased. But it's allowed to modify values when you
iterate on *keys*.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Ian Lee]

Cross posting the comment I’d left on the issue [1].

> My preference is to actually break that logic up and avoid the wrapping in 
> the first place, as in [2]. Which in this particular class has the side 
> benefit of that value being used again in the same function anyways.

> I'm starting to realize that Brandon Rhodes really had a big impact on my 
> ideas of styling as I've been learning Python these past few years, as this 
> was another one style I'm stealing from that same talk [3].

[1] http://bugs.python.org/msg263509 
[2] 
https://github.com/python/peps/commit/0c790e7b721bd13ad12ab9e6f6206836f398f9c4

~ Ian Lee | ianlee1...@gmail.com 
> On Apr 15, 2016, at 10:49, MRAB  wrote:
> 
> On 2016-04-15 18:03, Victor Stinner wrote:
> > Hum.
> >
> >  if (width == 0
> >  and height == 0
> >  and color == 'red'
> >  and emphasis == 'strong'
> >  or highlight > 100):
> >  raise ValueError("sorry, you lose")
> >
> > Please remove one space to vertically align "and" operators with the
> > opening parenthesis:
> >
> >  if (width == 0
> > and height == 0
> > and color == 'red'
> > and emphasis == 'strong'
> > or highlight > 100):
> >  raise ValueError("sorry, you lose")
> >
> > (I'm not sure that the difference is obvious in a mail client, you
> > need a fixed width font which is not the case in my Gmail editor.)
> >
> > It helps to visually see that the multiline test and the raise
> > instruction are in two different blocks.
> >
> > (Moreover, the pep8 checks of OpenStack simply reject such syntax, but
> > I cannot use this syntax anymore :-))
> >
> I always half-indent continuation lines:
> 
>if (width == 0
>  and height == 0
>  and color == 'red'
>  and emphasis == 'strong'
>  or highlight > 100):
>raise ValueError("sorry, you lose")
> ___
> Python-Dev mailing list
> Python-Dev@python.org 
> https://mail.python.org/mailman/listinfo/python-dev 
> 
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/ianlee1521%40gmail.com 
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Random832]

On Fri, Apr 15, 2016, at 16:41, Victor Stinner wrote:
> If the dictionary values are modified during the loop, the dict
> version is increased. But it's allowed to modify values when you
> iterate on *keys*.

Why is iterating over items different from iterating over keys?

in other words, why do I have to write:

for k in dict:
v = dict[k]
...do some stuff...
dict[k] = something

rather than

for k, v in dict.items():
...do some stuff...
dict[k] = something


It's not clear why the latter is something you want to prevent.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Ethan Furman]

On 04/15/2016 01:41 PM, Victor Stinner wrote:

2016-04-15 19:54 GMT+02:00 Jim J. Jewett:



(2)  Why *promise* not to update the version_tag when replacing a
value with itself?


It's an useful property. For example, let's say that you have a guard
on globals()['value']. The guard is created with value=3. An unit test
replaces the value with 50, but then restore the value to its previous
value (3). Later, the guard is checked to decide if an optimization
can be used.


I don't understand -- shouldn't the version be incremented with the 
value was replaced with 50, and again when re-replaced with 3?




(6)  I'm also not sure why version_tag *doesn't* solve the problem
of dicts that fool the iteration guards by mutating without changing
size ( https://bugs.python.org/issue19332 ) ... are you just saying
that the iterator views aren't allowed to rely on the version-tag
remaining stable, because replacing a value (as opposed to a
key-value pair) is allowed?


If the dictionary values are modified during the loop, the dict
version is increased. But it's allowed to modify values when you
iterate on *keys*.


I don't understand.  Could you provide a small example?

--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Victor Stinner]

2016-04-15 23:07 GMT+02:00 Random832 :
> Why is iterating over items different from iterating over keys?
>
> in other words, why do I have to write:
>
> for k in dict:
> v = dict[k]
> ...do some stuff...
> dict[k] = something
>
> rather than
>
> for k, v in dict.items():
> ...do some stuff...
> dict[k] = something
>
> It's not clear why the latter is something you want to prevent.

Hum, I think that you misunderstood what should be prevented. Please
see https://bugs.python.org/issue19332

Sorry, I don't know well this issue. I just know that sadly the PEP
509 doesn't help to fix this issue. Maybe it's not worth to mention
it...

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Victor Stinner]

2016-04-15 23:16 GMT+02:00 Ethan Furman :
>> It's an useful property. For example, let's say that you have a guard
>> on globals()['value']. The guard is created with value=3. An unit test
>> replaces the value with 50, but then restore the value to its previous
>> value (3). Later, the guard is checked to decide if an optimization
>> can be used.
>
> I don't understand -- shouldn't the version be incremented with the value
> was replaced with 50, and again when re-replaced with 3?

Oh wait, I'm tired and you are right.

Not increasing the value only helps on this code:

dict[key] = value
dict[key] = value  # version doesn't change


>> If the dictionary values are modified during the loop, the dict
>> version is increased. But it's allowed to modify values when you
>> iterate on *keys*.
>
> I don't understand.  Could you provide a small example?

For example, this loop is fine:

for key in dict:
   dict[key] = None

In this loop, the dict version is increased at each loop iteration.

For iter(dict), the check prevents a crash.

The following example raises a RuntimeError("dictionary changed size
during iteration"):

d={1:2}
for k in d:
d[k+1] = None

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Victor Stinner]

Hi,

FYI I updated the implementation of the PEP 509:
https://bugs.python.org/issue26058


2016-04-15 11:01 GMT+02:00 Antoine Pitrou :
> Why do this? It's a nice property that two dicts always have different
> version tags, and now you're killing this property for... no obvious
> reason?
>
> Do you really think dict.clear() is in need of micro-optimizing a
> couple CPU cycles away?

So, I played with Armin's idea. I confirm that it works for my use
case, guards on dict keys. It should also work on Yury's use case.

Antoine is right, it's really a micro-optimization. It shouldn't help
much for the integer overflow (which is not a real issue in practice).

I propose to leave the PEP unchanged to keep the nice property of
unique identifier for empty dictionaries. It can help for future use
cases.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Jim J. Jewett]

On Fri, Apr 15, 2016 at 4:41 PM, Victor Stinner
 wrote:
> 2016-04-15 19:54 GMT+02:00 Jim J. Jewett :

>> (2)  Why *promise* not to update the version_tag when replacing a
>> value with itself?

> It's an useful property. For example, let's say that you have a guard
> on globals()['value']. The guard is created with value=3. An unit test
> replaces the value with 50, but then restore the value to its previous
> value (3). Later, the guard is checked to decide if an optimization
> can be used.

> If the dict version is increased, you need a lookup. If the dict
> version is not increased, the guard is cheap.

I would expect the version to be increased twice, and therefore to
require a lookup.  Are you suggesting that unittest should provide an
example of resetting the version back to the original value when it
cleans up after itself?

> In C, it's very cheap to implement the test "new_value == old_value",
> it just compares two pointers.

Yeah, I understand that it is likely a win in terms of performance,
and a good way to start off (given that you're willing to do the
work).

I just worry that you may end up closing off even better optimizations
later, if you make too many promises about exactly how you will do
which ones.

Today, dict only cares about ==, and you (reasonably) think that full
== isn't always worth running ... but when it comes to which tests
*are* worth running, I'm not confident that the answers won't change
over the years.

>> [2A] Do you want to promise that replacing a value with a
>> non-identical object *will* trigger a version_tag update *even*
>> if the objects are equal?

> It's already written in the PEP:

I read that as a description of what the code does, rather than a spec
for what it should do... so it isn't clear whether I could count on
that remaining true.

For example, if I know that my dict values are all 4-digit integers,
can I write:

d[k]  = d[k] + 0

and be assured that the version_tag will bump?  Or is that something
that a future optimizer might optimize out?

>> (3)  It is worth being explicit on whether empty dicts can share
>> a version_tag of 0.  If this PEP is about dict content, then that
>> seems fine, and it may well be worth optimizing dict creation.

> This is not part of the PEP yet. I'm not sure that I will modify the
> PEP to use the version 0 for empty dictionaries. Antoine doesn't seem
> to be convinced :-)

True.  But do note that "not hitting the global counter an extra time
for every dict creation" is a more compelling reason than "we could
speed up dict.clear(), sometimes".


>> (4)  Please be explicit about the locking around version++; it
>> is enough to say that the relevant methods already need to hold
>> the GIL (assuming that is true).

> I don't think that it's important to mention it in the PEP. It's more
> an implementation detail. The version can be protected by atomic
> operations.

Now I'm the one arguing from a specific implementation.  :D

My thought was that any sort of locking (including atomic operations)
is slow, but if the GIL is already held, then there is no *extra*
locking cost. (Well, a slightly longer hold on the lock, but...)

>> (5)  I'm not sure I understand the arguments around a per-entry
>> version.

>> On the one hand, you never need a strong reference to the value;
>> if it has been collected, then it has obviously been removed from
>> the dict and should trigger a change even with per-dict.
>
> Let's say that you watch the key1 of a dict. The key2 is modified, it
> increases the version. Later, you test the guard: to check if the key1
> was modified, you need to lookup the key and compare the value. You
> need the value to compare it.

And the value for key1 is still there, so you can.

The only reason you would notice that the key2 value had gone away is
if you also care about key2 -- in which case the cached value is out
of date, regardless of what specific value it used to hold.

>> (6)  I'm also not sure why version_tag *doesn't* solve the problem
>> of dicts that fool the iteration guards by mutating without changing
>> size ( https://bugs.python.org/issue19332 ) ... are you just saying
>> that the iterator views aren't allowed to rely on the version-tag
>> remaining stable, because replacing a value (as opposed to a
>> key-value pair) is allowed?

> If the dictionary values are modified during the loop, the dict
> version is increased. But it's allowed to modify values when you
> iterate on *keys*.

Sure.  So?

I see three cases:

(A)  I don't care that the collection changed.  The python
implementation might, but I don't.  (So no bug even today.)

(B)  I want to process exactly the collection that I started with.  If
some of the values get replaced, then I want to complain, even if
python doesn't.  version_tag is what I want.

(C)  I want to process exactly the original keys, but go ahead and use
updated values.  The bug still bites, but ... I don't think this case
is any more common than B.

-jJ

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Victor Stinner]

.2016-04-15 23:45 GMT+02:00 Jim J. Jewett :
>> It's an useful property. For example, let's say that you have a guard
>> on globals()['value']. The guard is created with value=3. An unit test
>> replaces the value with 50, but then restore the value to its previous
>> value (3). Later, the guard is checked to decide if an optimization
>> can be used.
>
>> If the dict version is increased, you need a lookup. If the dict
>> version is not increased, the guard is cheap.
>
> I would expect the version to be increased twice, and therefore to
> require a lookup.  Are you suggesting that unittest should provide an
> example of resetting the version back to the original value when it
> cleans up after itself?

Sorry, as I wrote in another email that I was wrong. If you modify the
value, the version is increased. The discussed case is really a corner
case: the version does not change if the key is set again to exactly
the same value.

d[key] = value
d[key] = value

It's just that it's cheap to implement it :-)


>> In C, it's very cheap to implement the test "new_value == old_value",
>> it just compares two pointers.
>
> Yeah, I understand that it is likely a win in terms of performance,
> and a good way to start off (given that you're willing to do the
> work).
>
> I just worry that you may end up closing off even better optimizations
> later, if you make too many promises about exactly how you will do
> which ones.
>
> Today, dict only cares about ==, and you (reasonably) think that full
> == isn't always worth running ... but when it comes to which tests
> *are* worth running, I'm not confident that the answers won't change
> over the years.

I checked, currently there is no unit test for a==b, only for a is b.
I will add add a test for a==b but a is not b, and ensure that the
version is increased.


>>> [2A] Do you want to promise that replacing a value with a
>>> non-identical object *will* trigger a version_tag update *even*
>>> if the objects are equal?
>
>> It's already written in the PEP:
>
> I read that as a description of what the code does, rather than a spec
> for what it should do... so it isn't clear whether I could count on
> that remaining true.
>
> For example, if I know that my dict values are all 4-digit integers,
> can I write:
>
> d[k]  = d[k] + 0
>
> and be assured that the version_tag will bump?  Or is that something
> that a future optimizer might optimize out?

Hum, I will try to clarify that.


>>> (4)  Please be explicit about the locking around version++; it
>>> is enough to say that the relevant methods already need to hold
>>> the GIL (assuming that is true).
>
>> I don't think that it's important to mention it in the PEP. It's more
>> an implementation detail. The version can be protected by atomic
>> operations.
>
> Now I'm the one arguing from a specific implementation.  :D
>
> My thought was that any sort of locking (including atomic operations)
> is slow, but if the GIL is already held, then there is no *extra*
> locking cost. (Well, a slightly longer hold on the lock, but...)

Hum, since the PEP clarify targets CPython, I will simply described
its implementation, so explain that the GIL ensures that version++ is
atomic.


>>> On the one hand, you never need a strong reference to the value;
>>> if it has been collected, then it has obviously been removed from
>>> the dict and should trigger a change even with per-dict.
>>
>> Let's say that you watch the key1 of a dict. The key2 is modified, it
>> increases the version. Later, you test the guard: to check if the key1
>> was modified, you need to lookup the key and compare the value. You
>> need the value to compare it.
>
> And the value for key1 is still there, so you can.

Sorry, how do you want to compare that dict[key1] value didn't change,
using the value identifier? dict[key1] is old_value_id?

The problem with storing an identifier (a pointer in C) with no strong
reference is when the object is destroyed, a new object can likely get
the same identifier. So it's likely that "dict[key] is old_value_id"
can be true even if dict[key] is now a new object.


> The only reason you would notice that the key2 value had gone away is
> if you also care about key2 -- in which case the cached value is out
> of date, regardless of what specific value it used to hold.

I don't understand, technically, what do you mean by "out of date" for
an object?


>> If the dictionary values are modified during the loop, the dict
>> version is increased. But it's allowed to modify values when you
>> iterate on *keys*.
>
> Sure.  So?
>
> I see three cases:
>
> (A)  I don't care that the collection changed.  The python
> implementation might, but I don't.  (So no bug even today.)

I'm sorry, I don't understand your description. What do you mean by
"collection"? It's different if you modify dict *keys*, or dict
*values*, or both.

Serhiy opened an issue because he wants to raise an exception if keys
are modified while you iterate on keys:
https://bugs.

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Peter Ludemann via Python-Dev]

If Python ever adopts the BCPL rule for implicit line continuation if the
last thing on a line is an operator (or if there's an open parentheses),
then the break-after-an-operator rule would be more persuasive.
;)

[IIRC, the BCPL rule was that there was an implicit continuation if the
grammar would not allow inserting a semicolon at the end of the line, which
covered both the open-parens and last-item-is-operator cases, and probably
a few others.] But I should shut up and leave shut discussions to
python-ideas.


On 15 April 2016 at 13:48, Ian Lee  wrote:

> Cross posting the comment I’d left on the issue [1].
>
> > My preference is to actually break that logic up and avoid the wrapping
> in the first place, as in [2]. Which in this particular class has the side
> benefit of that value being used again in the same function anyways.
>
> > I'm starting to realize that Brandon Rhodes really had a big impact on
> my ideas of styling as I've been learning Python these past few years, as
> this was another one style I'm stealing from that same talk [3].
>
> [1] http://bugs.python.org/msg263509
> [2]
> https://github.com/python/peps/commit/0c790e7b721bd13ad12ab9e6f6206836f398f9c4
>
> ~ Ian Lee | ianlee1...@gmail.com
>
> On Apr 15, 2016, at 10:49, MRAB  wrote:
>
> On 2016-04-15 18:03, Victor Stinner wrote:
> > Hum.
> >
> >  if (width == 0
> >  and height == 0
> >  and color == 'red'
> >  and emphasis == 'strong'
> >  or highlight > 100):
> >  raise ValueError("sorry, you lose")
> >
> > Please remove one space to vertically align "and" operators with the
> > opening parenthesis:
> >
> >  if (width == 0
> > and height == 0
> > and color == 'red'
> > and emphasis == 'strong'
> > or highlight > 100):
> >  raise ValueError("sorry, you lose")
> >
> > (I'm not sure that the difference is obvious in a mail client, you
> > need a fixed width font which is not the case in my Gmail editor.)
> >
> > It helps to visually see that the multiline test and the raise
> > instruction are in two different blocks.
> >
> > (Moreover, the pep8 checks of OpenStack simply reject such syntax, but
> > I cannot use this syntax anymore :-))
> >
> I always half-indent continuation lines:
>
>if (width == 0
>  and height == 0
>  and color == 'red'
>  and emphasis == 'strong'
>  or highlight > 100):
>raise ValueError("sorry, you lose")
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ianlee1521%40gmail.com
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/pludemann%40google.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 8 updated on whether to break before or after a binary update

[Random832]

On Fri, Apr 15, 2016, at 23:46, Peter Ludemann via Python-Dev wrote:
> If Python ever adopts the BCPL rule for implicit line continuation if
> the last thing on a line is an operator (or if there's an open
> parentheses), then the break-after-an-operator rule would be more
> persuasive. ;)
>
> [IIRC, the BCPL rule was that there was an implicit continuation if
> the grammar would not allow inserting a semicolon at the end of the
> line, which covered both the open-parens and last-item-is-operator
> cases, and probably a few others.] But I should shut up and leave shut
> discussions to python-ideas.

Sounds like Visual Basic. Meanwhile, Javascript's rule is that there's
an implicit semicolon if and only if the grammar would not allow the
two lines to be considered as a single statement. Insanity comes in
all flavors.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com