Re: [Python-Dev] Tracker Stats

[R. David Murray]

On Mon, 23 Jun 2014 17:52:33 +0200, francis  wrote:
> 
> > Hi,
> > I added a new "stats" page to the bug tracker:
> > http://bugs.python.org/issue?@template=stats
> Thanks Ezio,
> 
> Two questions:
> how hard would be to add (or enhance) a chart with the
> “open issues type enhancement” and “open issues type bug”
> info ?
> 
> In the summaries there is a link to “Issues with patch”,
> means that the ones not listed there are in “needs patch”
> or “new” status?

The stats graphs are based on the data generated for the
weekly issue report.  I have a patched version of that
report that adds the bug/enhancement info.  I'll try to dig
it up this week; someone ping me if I forget :)  It think
the patch will need to be updated based on Ezio's changes.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker Stats

[R. David Murray]

On Tue, 24 Jun 2014 20:43:41 +0200, francis  wrote:
> On 06/24/2014 03:50 AM, Ezio Melotti wrote:
> >>From the first graph you can see that out of the 4500+ open issues,
> > about 2000 have a patch.
> One would like to start with the ones that are bugs ;-) and see some
> status line trying to drop to 0 (is that possible :-) ?)
> 
> > We need more reviewers and committers :)
> more patch writers: yes,
> more patch reviewers: yes,

Anyone can review patches, in case that isn't clear.

> more committers: ?? automate!! :-)

That's a goal of the python-workflow interest group.  Unfortunately
between billable work and GSOC mentoring I haven't had time to do much
there lately.  Our first goal is to make the review step easier to manage
(know which patches really need review, be able to list patches where
community review is thought to be complete) by improving the tracker,
then we'll look at creating the patch gating system Nick has talked
about previously.  Still needs a committer to approve the patch, but it
should increase the throughput considerably.

In the meantime, something that would help would be if people would do
reviews and say on the issue "I think this is commit ready" and have
the issue moved to 'commit review' stage.  Do that a few times where
people who are already triagers/committers agree with you, and you'll
get triage privileges on the tracker.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bluetooth 4.0 support in "socket" module

[R. David Murray]

On Mon, 14 Jul 2014 16:42:25 -0400, Terry Reedy  wrote:
> On 7/14/2014 9:57 AM, Tim Tisdall wrote:
> 
> 2 questions not answered yet.
> 
> > Also, is there a method to test changes against all the different *nix
> > variations?
> 
> We have a set of buildbots.
> https://www.python.org/dev/buildbot/
> 
> > Is Bluez the standard across the different *nix variations?
> 
> No idea.

It would be really nice to answer that and the related testing questions.
The socket module has bluetooth support, but there are no tests.
An effort to write some was started at the Bloomberg sprint last month,
but nothing has been posted to the issue yet:

http://bugs.python.org/issue7687

Is Bluetooth 4.0 something different from what the socket module already
has?

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another case for frozendict

[R. David Murray]

On Wed, 16 Jul 2014 03:27:23 +0100, MRAB  wrote:
> Here's another use-case.
> 
> Using the 're' module:
> 
>  >>> import re
>  >>> # Make a regex.
> ... p = re.compile(r'(?P\w+)\s+(?P\w+)')
>  >>>
>  >>> # What are the named groups?
> ... p.groupindex
> {'first': 1, 'second': 2}
>  >>>
>  >>> # Perform a match.
> ... m = p.match('FIRST SECOND')
>  >>> m.groupdict()
> {'first': 'FIRST', 'second': 'SECOND'}
>  >>>
>  >>> # Try modifying the pattern object.
> ... p.groupindex['JUNK'] = 'foobar'
>  >>>
>  >>> # What are the named groups now?
> ... p.groupindex
> {'first': 1, 'second': 2, 'JUNK': 'foobar'}
>  >>>
>  >>> # And the match object?
> ... m.groupdict()
> Traceback (most recent call last):
>File "", line 2, in 
> IndexError: no such group
> 
> It can't find a named group called 'JUNK'.

IMO, preventing someone from shooting themselves in the foot by modifying
something they shouldn't modify according to the API is not a Python
use case ("consenting adults").

> And with a bit more tinkering it's possible to crash Python. (I'll
> leave that as an exercise for the reader! :-))

Preventing a Python program from being able to crash the interpreter,
that's a use case :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another case for frozendict

[R. David Murray]

On Wed, 16 Jul 2014 03:27:23 +0100, MRAB  wrote:
>  >>> # Try modifying the pattern object.
> ... p.groupindex['JUNK'] = 'foobar'
>  >>>
>  >>> # What are the named groups now?
> ... p.groupindex
> {'first': 1, 'second': 2, 'JUNK': 'foobar'}
>  >>>
>  >>> # And the match object?
> ... m.groupdict()
> Traceback (most recent call last):
>File "", line 2, in 
> IndexError: no such group
> 
> It can't find a named group called 'JUNK'.

After I hit send on my previous message, I thought more about your
example.  One of the issues here is that modifying the dict breaks an
invariant of the API.  I have a similar situation in the email module,
and I used the same solution you did in regex: always return a new dict.
It would be nice to be able to return a frozendict instead of having the
overhead of building a new dict on each call.  That by itself might not
be enough reason.  But, if the user wants to use the data in modified form
elsewhere, they would then have to construct a new regular dict out of it,
making the decision to vary the data from what matches the state of the
object it came from an explicit one.  That seems to fit the Python zen
("explicit is better than implicit").

So I'm changing my mind, and do consider this a valid use case, even
absent the crash.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another case for frozendict

[R. David Murray]

On Wed, 16 Jul 2014 14:04:29 -, dw+python-...@hmmz.org wrote:
> On Wed, Jul 16, 2014 at 09:47:59AM -0400, R. David Murray wrote:
> 
> > It would be nice to be able to return a frozendict instead of having the
> > overhead of building a new dict on each call.
> 
> There already is an in-between available both to Python and C:
> PyDictProxy_New() / types.MappingProxyType. It's a one line change in
> each case to return a temporary intermediary, using something like (C):
> Py_INCREF(self->dict)
> return self->dict;
> 
> To
> return PyDictProxy_New(self->dict);
> 
> Or Python:
> return self.dct
> 
> To
> return types.MappingProxyType(self.dct)
> 
> Which is cheaper than a copy, and avoids having to audit every use of
> self->dict to ensure the semantics required for a "frozendict" are
> respected, i.e. no mutation occurs after the dict becomes visible to the
> user, and potentially has __hash__ called.
> 
> 
> > That by itself might not be enough reason.  But, if the user wants to
> > use the data in modified form elsewhere, they would then have to
> > construct a new regular dict out of it, making the decision to vary
> > the data from what matches the state of the object it came from an
> > explicit one.  That seems to fit the Python zen ("explicit is better
> > than implicit").
> > 
> > So I'm changing my mind, and do consider this a valid use case, even
> > absent the crash.
> 
> Avoiding crashes seems a better use for a read-only proxy, rather than a
> hashable immutable type.

Good point.  MappingProxyType wasn't yet exposed when I wrote that email
code.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another case for frozendict

[R. David Murray]

On Wed, 16 Jul 2014 10:10:07 -0700, Devin Jeanpierre  
wrote:
> On Wed, Jul 16, 2014 at 6:37 AM, R. David Murray  
> wrote:
> > IMO, preventing someone from shooting themselves in the foot by modifying
> > something they shouldn't modify according to the API is not a Python
> > use case ("consenting adults").
> 
> Then why have immutable objects at all? Why do you have to put tuples
> and frozensets inside sets, instead of lists and sets? Compare with
> Java, which really is "consenting adults" here -- you can add a
> mutable object to a set, just don't mutate it, or you might not be
> able to find it in the set again.
> 
> Several people seem to act as if the Pythonic way is to not allow for
> any sort of immutable types at all. ISTM people are trying to
> retroactively claim some standard of Pythonicity that never existed.
> Python can and does protect you from shooting yourself in the foot by
> making objects immutable. Or do you have another explanation for the
> proliferation of immutable types, and the inability to add mutable
> types to sets and dicts?
> 
> Using a frozendict to protect and enforce an invariant in the re
> module is entirely reasonable. So is creating a new dict each time.
> The intermediate -- reusing a mutable dict and failing in
> incomprehensible ways if you mutate it, and potentially even crashing
> due to memory safety issues -- is not Pythonic at all.

You'll note I ended up agreeing with you there: when mutation breaks an
invariant of the object it came from, that's an issue.  Which would be
the case if you could use mutable objects as keys.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

[R. David Murray]

On Sun, 10 Aug 2014 13:57:36 +1000, Nick Coghlan  wrote:
> On 10 August 2014 13:20, Antoine Pitrou  wrote:
> > Le 09/08/2014 12:43, Ben Hoyt a écrit :
> >
> >> Just thought I'd share some of my excitement about how fast the all-C
> >> version [1] of os.scandir() is turning out to be.
> >>
> >> Below are the results of my scandir / walk benchmark run with three
> >> different versions. I'm using an SSD, which seems to make it
> >> especially faster than listdir / walk. Note that benchmark results can
> >> vary a lot, depending on operating system, file system, hard drive
> >> type, and the OS's caching state.
> >>
> >> Anyway, os.walk() can be FIFTY times as fast using os.scandir().
> >
> >
> > Very nice results, thank you :-)
> 
> Indeed!
> 
> This may actually motivate me to start working on a redesign of
> walkdir at some point, with scandir and DirEntry objects as the basis.
> My original approach was just too slow to be useful in practice (at
> least when working with trees on the scale of a full Fedora or RHEL
> build hosted on an NFS share).

There is another potentially good place in the stdlib to apply scandir:
iglob.  See issue 22167.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[R. David Murray]

On Sun, 10 Aug 2014 13:12:26 -0700, Glenn Linderman  
wrote:
> On 8/10/2014 1:24 AM, Stephen J. Turnbull wrote:
> > Actually ... if I were a fan of the "".join() idiom, I'd seriously
> > propose 0.sum(numeric_iterable) as the RightThang{tm].  Then we could
> > deprecate "".join(string_iterable) in favor of "".sum(string_iterable)
> > (with the same efficient semantics).
> Actually, there is no need to wait for 0.sum() to propose "".sum... but 
> it is only a spelling change, so no real benefit.
> 
> Thinking about this more, maybe it should be a class function, so that 
> it wouldn't require an instance:
> 
> str.sum( iterable_containing_strings )
> 
> [ or  str.join( iterable_containing_strings ) ]

That's how it used to be spelled in python2.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[R. David Murray]

On Sun, 10 Aug 2014 13:12:26 -0700, Glenn Linderman  
wrote:
> On 8/10/2014 1:24 AM, Stephen J. Turnbull wrote:
> > Actually ... if I were a fan of the "".join() idiom, I'd seriously
> > propose 0.sum(numeric_iterable) as the RightThang{tm].  Then we could
> > deprecate "".join(string_iterable) in favor of "".sum(string_iterable)
> > (with the same efficient semantics).
> Actually, there is no need to wait for 0.sum() to propose "".sum... but 
> it is only a spelling change, so no real benefit.
> 
> Thinking about this more, maybe it should be a class function, so that 
> it wouldn't require an instance:
> 
> str.sum( iterable_containing_strings )
> 
> [ or  str.join( iterable_containing_strings ) ]

Sorry, I mean 'string.join' is how it used to be spelled.  Making it a
class method is indeed slightly different.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python2.7 infinite recursion when loading pickled object

[R. David Murray]

On Mon, 11 Aug 2014 21:43:00 +1000, Chris Angelico  wrote:
> On Mon, Aug 11, 2014 at 9:40 PM, Peter Otten <__pete...@web.de> wrote:
> > Read again. The OP tries to delegate attribute lookup to an (existing)
> > attribute.
> >
> > IMO the root cause of the problem is that pickle looks up __dunder__ methods
> > in the instance rather than the class.
> 
> The recursion comes from the attempted lookup of self.item, when
> __init__ hasn't been called.

Indeed, and this is what the OP missed.  With a class like this, it is
necessary to *make* it pickleable, since the pickle protocol doesn't
call __init__.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[R. David Murray]

On Sat, 23 Aug 2014 00:21:18 +0200, Oleg Broytman  wrote:
>I'm involved in developing and maintaining a few big commercial
> projects that will hardly be ported to Python3. So I'm stuck with
> Python2 for many years and I haven't tried Python3. May be I should try
> a small personal project, but certainly not this year. May be the next
> one...

Yes, you should try it.  Really, it's not the monster you are
constructing in your mind.  The functions that read filenames and return
them as text use surrogate escape to preserve the bytes, and the
functions that accept filenames use surrogate escape to recover those
bytes before passing them back to the OS.  So posix binary filenames
just work, as long as the only thing you depend on is being able to
split and join them on the / character (and possibly the . character)
and otherwise treat the names as black boxes...which is exactly the same
situation you are in in python2.

If you need to read filenames out of a file, you'll need to specify the
surrogate escape error handler so that the bytes will be there to be
recovered when you pass them to the file system functions, but it will
work.

Or, as discussed, you can treat them as binary and use the os level
functions that accept binary input (which are exactly the ones you are
used to using in python2).  This includes os.path.split and
os.path.join, which as noted are the only things you can depend on
working correctly when you don't know the encoding of the filenames.

So, the way to look at this is that python3 is no worse[1] than python2 for
handling posix binary filenames, and also provides additional features
if you *do* know the correct encoding of the filenames.

--David

[1] modulo any remaining API bugs, which is exactly where this thread
started: trying to figure out which APIs need to be able to handle
binary paths and/or surrogate escaped paths so that posix filenames
consistently work as well in python3 as they did in python2).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[R. David Murray]

On Sat, 23 Aug 2014 21:08:29 +1000, Steven D'Aprano  wrote:
> When I started this email, I originally began to say that the actual 
> problem was with byte file names that cannot be decoded into Unicode 
> using the system encoding (typically UTF-8 on Linux systems. But I've 
> actually had difficulty demonstrating that it actually is a problem. I 
> started with a byte sequence which is invalid UTF-8, namely:
> 
> b'ZZ\xdb\xdf\xfa\xff'
> 
> created a file with that name, and then tried listing it with 
> os.listdir. Even in Python 3.1 it worked fine. I was able to list the 
> directory and open the file, so I'm not entirely sure where the problem 
> lies exactly. Can somebody demonstrate the failure mode?

The "failure" happens only when you try to cross from the domain of
posix binary filenames into the domain of text streams (that is, a
stream with a consistent encoding).  If you stick with os interfaces
that handle filenames, Python3 handles posix bytes filenames just fine
(though there may be a few corner-case rough edges yet to be fixed, and
the standard streams was one of them).

The difficultly comes if you try to use a filename that contains
undecodable bytes in a non-os-interface text-context (such as writing it
to a text file that you have declared to be a utf-8 encoding): there you
will get an error...not completely unlike the old "your code works until
your user uses unicode" problem we had in python2, but in this case only
happening in a very narrow set of circumstances involving trying to
translate between one domain (posix binary filenames) and another domain
(io streams with a consistent declared encoding).  This is not a common
operation, but appears to be the one Oleg is concerned about. The old
unicode-blowup errors would happen almost any time someone with a
non-ascii language tried to use a program written by an ascii-only
programmer (which was most of us).

The same problem existed in python2 if your goal was to produce a stream
with a consistent encoding, but now python3 treats that as an error.  If
you really want a stream with an inconsistent encoding, open it as
binary and use the surrogate escape error handler to recover the bytes
in the filenames.  That is, *be explicit* about your intentions.

So yes, we've shifted a burden from those who want non-ascii text to
work consistently to those who wanted inconsistently encoded text to "just
work" (or rather *appear* to "just work").  The number of people who
benefit from the improved text model *greatly* outweighs the number of
people inconvenienced by the new strictness when the domain line (posix
binary filenames to consistently encoded text stream) are crossed.  And
the result is more *valid* programs, and fewer unexpected errors
overall, with no inconvenience unless that domain line is crossed,
and even then the inconvenience is limited to the open call that creates
the binary stream.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[R. David Murray]

On Sat, 23 Aug 2014 19:33:06 +0300, Marko Rauhamaa  wrote:
> "R. David Murray" :
> 
> > The same problem existed in python2 if your goal was to produce a stream
> > with a consistent encoding, but now python3 treats that as an error.
> 
> I have a different interpretation of the situation: as a rule, use byte
> strings in Python3. Text strings are a special corner case for
> applications that have to deal with human languages.

Clearly, then, you are writing unix (or perhaps posix)-only programs.

Also, as has been discussed in this thread previously, any program that
deals with filenames is dealing with human readable languages, even
if posix itself treats the filenames as bytes.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[R. David Murray]

On Tue, 26 Aug 2014 11:25:19 +0900, "Stephen J. Turnbull"  
wrote:
> R. David Murray writes:
> 
>  > Also, as has been discussed in this thread previously, any program that
>  > deals with filenames is dealing with human readable languages, even
>  > if posix itself treats the filenames as bytes.
> 
> That's a bit extreme.  I can name two interesting applications
> offhand: git's object database and the Coda filesystem's containers.

As soon as I hit send I realized there were a few counter examples :)
So, replace "any" with "most".

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[R. David Murray]

On Sun, 24 Aug 2014 13:27:55 +1000, Nick Coghlan  wrote:
> As some examples of where bilingual computing breaks down:
> 
> * My NFS client and server may have different locale settings
> * My FTP client and server may have different locale settings
> * My SSH client and server may have different locale settings
> * I save a file locally and send it to someone with a different locale setting
> * I attempt to access a Windows share from a Linux client (or vice-versa)
> * I clone my POSIX hosted git or Mercurial repository on a Windows client
> * I have to connect my Linux client to a Windows Active Directory
> domain (or vice-versa)
> * I have to interoperate between native code and JVM code
> 
> The entire computing industry is currently struggling with this
> monolingual (ASCII/Extended ASCII/EBCDIC/etc) -> bilingual (locale
> encoding/code pages) -> multilingual (Unicode) transition. It's been
> going on for decades, and it's still going to be quite some time
> before we're done.
> 
> The POSIX world is slowly clawing its way towards a multilingual model
> that actually works: UTF-8
> Windows (including the CLR) and the JVM adopted a different
> multilingual model, but still one that actually works: UTF-16-LE

This kind of puts the "length" of the python2->python3 transition
period in perspective, doesn't it?

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path related questions for Guido

[R. David Murray]

On Thu, 28 Aug 2014 10:15:40 -0700, Glenn Linderman  
wrote:
> On 8/28/2014 12:30 AM, MRAB wrote:
> > On 2014-08-28 05:56, Glenn Linderman wrote:
> >> On 8/27/2014 6:08 PM, Stephen J. Turnbull wrote:
> >>> Glenn Linderman writes:
> >>>   > On 8/26/2014 4:31 AM, MRAB wrote:
> >>>   > > On 2014-08-26 03:11, Stephen J. Turnbull wrote:
> >>>   > >> Nick Coghlan writes:
> >>>
> >>>   > > How about:
> >>>   > >
> >>>   > > replace_surrogate_escapes(s, replacement='\uFFFD')
> >>>   > >
> >>>   > > If you want them removed, just pass an empty string as the
> >>>   > > replacement.
> >>>
> >>> That seems better to me (I had too much C for breakfast, I think).
> >>>
> >>>   > And further, replacement could be a vector of 128 characters, to do
> >>>   > immediate transcoding,
> >>>
> >>> Using what encoding?
> >>
> >> The vector would contain the transcoding. Each lone surrogate would map
> >> to a character in the vector.
> >>
> >>> If you knew that much, why didn't you use
> >>> (write, if necessary) an appropriate codec?  I can't envision this
> >>> being useful.
> >>
> >> If the data format describes its encoding, possibly containing data from
> >> several encodings in various spots, then perhaps it is best read as
> >> binary, and processed as binary until those definitions are found.
> >>
> >> But an alternative would be to read with surrogate escapes, and then
> >> when the encoding is determined, to transcode the data. Previously, a
> >> proposal was made to reverse the surrogate escapes to the original
> >> bytes, and then apply the (now known) appropriate codec. There are not
> >> appropriate codecs that can convert directly from surrogate escapes to
> >> the desired end result. This technique could be used instead, for
> >> single-byte, non-escaped encodings. On the other hand, writing specialty
> >> codecs for the purpose would be more general.
> >>
> > There'll be a surrogate escape if a byte couldn't be decoded, but just
> > because a byte could be decoded, it doesn't mean that it's correct.
> >
> > If you picked the wrong encoding, the other codepoints could be wrong
> > too.
> 
> Aha! Thanks for pointing out the flaw in my reasoning. But that means it 
> is also pretty useless to "replace_surrogate_escapes" at all, because it 
> only cleans out the non-decodable characters, not the incorrectly 
> decoded characters.

Well, replace would still be useful for ASCII+surrogateescape.  Also for
cases where the data stream is *supposed* to be in a given encoding, but
contains undecodable bytes.  Showing the stuff that incorrectly decodes
as whatever it decodes to is generally what you want in that case.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path related questions for Guido

[R. David Murray]

On Thu, 28 Aug 2014 10:54:44 -0700, Glenn Linderman  
wrote:
> On 8/28/2014 10:41 AM, R. David Murray wrote:
> > On Thu, 28 Aug 2014 10:15:40 -0700, Glenn Linderman  
> > wrote:
> >> On 8/28/2014 12:30 AM, MRAB wrote:
> >>> There'll be a surrogate escape if a byte couldn't be decoded, but just
> >>> because a byte could be decoded, it doesn't mean that it's correct.
> >>>
> >>> If you picked the wrong encoding, the other codepoints could be wrong
> >>> too.
> >> Aha! Thanks for pointing out the flaw in my reasoning. But that means it
> >> is also pretty useless to "replace_surrogate_escapes" at all, because it
> >> only cleans out the non-decodable characters, not the incorrectly
> >> decoded characters.
> > Well, replace would still be useful for ASCII+surrogateescape.
> 
> How?

Because there "can't" be any incorrectly decoded bytes in the ASCII part,
so all undecodable bytes turning into 'unrecognized character' glyphs
is useful. "can't" is in quotes because of course if you decode random
binary data as ASCII+surrogate escape you could get a mess just like any
other encoding, so this is really a "more *likely* to be useful" version
of my second point, because "real" ASCII with some junk bytes mixed in
is much more likely to be encountered in the wild than, say, utf-8 with
some junk bytes mixed in (although is probably changing as use of utf-8
becomes more widespread, so this point applies to utf-8 as well).

> > Also for
> > cases where the data stream is *supposed* to be in a given encoding, but
> > contains undecodable bytes.  Showing the stuff that incorrectly decodes
> > as whatever it decodes to is generally what you want in that case.
>
> Sure, people can learn to recognize mojibake for what it is, and maybe 
> even learn to recognize it for what it was intended to be, in limited 
> domains. But suppressing/replacing the surrogates doesn't help with 

Well, it does if the alternative is not being able to display the string
to the user at all.  And yeah, people being able to recognize mojibake
in specific problem domains is what I'm talking about...not perhaps a
great use case, but it is a use case.

> that... would it not be better to replace the surrogates with an escape 
> sequence that shows the original, undecodable, byte value?  Like  \xNN ?

Yeah, that idea has been floated as well, and I think it would indeed be
more useful than the 'unknown character' glyph.  I've also seen fonts
that display the hex code inside a box character when the code point is
unknown, which would be cool...but that can hardly be part of unicode,
can it? :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Fri, 29 Aug 2014 17:11:35 -0400, Donald Stufft  wrote:
> Sorry I was on my phone and didn’t get to fully reply to this.
> > On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg  wrote:
> > 
> > * configuration:
> > 
> >   It would be good to be able to switch this on or off
> >   without having to change the code, e.g. via a command
> >   line switch and environment variable; perhaps even
> >   controlling whether or not to raise an exception or
> >   warning.
> 
> I’m on the fence about this, if someone provides a certificate
> that we can validate against (which can be done without
> touching the code) then the only thing that really can’t be
> “fixed” without touching the code is if someone has a certificate
> that is otherwise invalid (expired, not yet valid, wrong hostname,
> etc). I’d say if I was voting on this particular thing I’d be -0, I’d
> rather it didn’t exist but I wouldn’t cry too much if it did.

Especially if you want an accelerated change, there must be a way to
*easily* get back to the previous behavior, or we are going to catch a
lot of flack.  There may be only 7% of public certs that are problematic,
but I'd be willing to bet you that there are more not-really-public ones
that are critical to day to day operations *somewhere* :)

wget and curl have 'ignore validation' as a command line flag for a reason.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Fri, 29 Aug 2014 18:00:50 -0400, Donald Stufft  wrote:
> 
> On Aug 29, 2014, at 5:42 PM, R. David Murray  wrote:
> > Especially if you want an accelerated change, there must be a way to
> > *easily* get back to the previous behavior, or we are going to catch a
> > lot of flack.  There may be only 7% of public certs that are problematic,
> > but I'd be willing to bet you that there are more not-really-public ones
> > that are critical to day to day operations *somewhere* :)
> > 
> > wget and curl have 'ignore validation' as a command line flag for a reason.
> > 
> 
> Right, that’s why I’m on the fence :)
> 
> On one hand, it’s going to break things for some people, (arguably they are
> already broken, just silently so, but we’ll leave that argument aside) and a
> way to get back the old behavior is good. There are already ways within
> the Python code itself, so that’s covered. From outside of the Python code
> there are ways if the certificate is untrusted but otherwise valid which are
> pretty easy to do. The major “gap” is when you have an actual invalid
> certificate due to expiration or hostname or some other such thing.
> 
> On the other hand Python is not wget/curl and the people who are most
> likely to be the target for a “I can’t change the code but I need to get 
> the
> old behavior back” are people who are likely to not be invoking Python
> itself but using something written in Python which happens to be using
> Python. IOW they might be executing “foobar” not “python -m foobar”.

Right, so an environment variable is better than a command line switch,
for Python.

> Like I said though, I’m personally fine either way so don’t take this as
> being against that particular change!

Ack.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Sat, 30 Aug 2014 14:03:57 +0200, "M.-A. Lemburg"  wrote:
> On 30.08.2014 12:55, Antoine Pitrou wrote:
> > On Sat, 30 Aug 2014 12:46:47 +0200
> > "M.-A. Lemburg"  wrote:
> >>> That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE
> >>> env vars (or, better, by specific settings *inside* the application).
> >>>
> >>> I'm against multiplying environment variables, as it makes it more
> >>> difficult to assess the actual security of a setting. The danger of an
> >>> ill-secure setting is much more severe than with hash randomization.
> >>
> >> You have a point there. So how about just a python run-time switch
> >> and no env var ?
> > 
> > Well, why not, but does it have a value over letting the code properly
> > configure their SSLContext?
> 
> Yes, because when Python changes the default to be validating
> and more secure, application developers will do the same as
> they do now: simply use the defaults ;-)

But neither of those addresses the articulated use case: someone *using*
a program implemented in python that does not itself provide a way to
disable the new default security (because it is *new*).  Only an
environment variable will do that.

Since the environment variable is opt-in, I think the "consenting
adults" argument applies to Alex's demure about "multiple connections".
It could still emit the warnings.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Sun, 31 Aug 2014 03:25:25 +0200, Antoine Pitrou  wrote:
> On Sun, 31 Aug 2014 09:26:30 +1000
> Nick Coghlan  wrote:
> > >>
> > >>   * configuration:
> > >>
> > >> It would be good to be able to switch this on or off
> > >> without having to change the code, e.g. via a command
> > >> line switch and environment variable; perhaps even
> > >> controlling whether or not to raise an exception or
> > >> warning.
> > >>
> > >>   * choice of trusted certificate:
> > >>
> > >> Instead of hard wiring using the system CA roots into
> > >> Python it would be good to just make this default and
> > >> permit the user to point Python to a different set of
> > >> CA roots.
> > >>
> > >> This would enable using self signed certs more easily.
> > >> Since these are often used for tests, demos and education,
> > >> I think it's important to allow having more control of
> > >> the trusted certs.
> > >
> > >
> > > +1 for PEP with above changes.
> > 
> > Ditto from me.
> > 
> > In relation to changing the Python CLI API to offer some of the wget/curl
> > style command line options, I like the idea of providing recipes in the
> > docs for implementing them at the application layer, but postponing making
> > the *default* behaviour configurable that way.
> 
> I'm against any additional environment variables and command-line
> options. It will only complicate and obscure the security parameters of
> certificate validation.
> 
> The existing knobs have already been mentioned in this thread, I won't
> mention them here again.

Do those knobs allow one to instruct urllib to accept an invalid
certificate without changing the program code?

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Sun, 31 Aug 2014 16:45:42 +1000, Nick Coghlan  wrote:
> On 31 August 2014 16:16, Donald Stufft  wrote:
> >
> > On Aug 31, 2014, at 2:09 AM, Nick Coghlan  wrote:
> >
> > At the same time, we need to account for the fact that most existing
> > organisations still trust in perimeter defence for their internal
> > network security, and hence tolerate (or even actively encourage) the
> > use of unsecured connections, or skipping certificate validation,
> > internally. This is actually a really terrible idea, but it's still
> > incredibly common due to the general failure of the technology
> > industry to take usability issues seriously when we design security
> > systems (at least until recently) - doing the wrong "unsafe" thing is
> > genuinely easier than doing things right.
> >
> >
> > Just a quick clarification in order to be a little clearer, this change will
> > (obviously) only effect those who trust perimeter security *and* decided to
> > install an invalid certificate instead of just using HTTP. I'm not saying
> > that
> > this doesn't happen, just being specific (I'm not actually sure why they
> > would
> > install a TLS certificate at all if they are trusting perimeter security,
> > but
> > I'm sure folks do).
> 
> It's the end result when a company wide edict to use HTTPS isn't
> backed up by the necessary documentation and training on how to get a
> properly signed cert from your internal CA (or, even better, when such
> an edict comes down without setting up an internal CA first). Folks
> hit the internet instead, find instructions on creating a self-signed
> cert, install that, and tell their users to ignore the security
> warning and accept the cert. Historically, Python clients have "just
> worked" in environments that required a click-through on the browser
> side, since you had to opt in to checking the certificates properly.
> 
> Self-signed certificates can also be really handy for doing local
> testing - you're not really aiming to authenticate the connection in
> that case, you're just aiming to test that the secure connection
> machinery is all working properly.

Self-signed certificates are not crazy in an internal corporate
environment even when properly playing the defense in depth game.  Once
you've acked the cert the first time, you will be warned if it changes
(like an ssh host key).  Sure, as Nick says the corp could set up an
internal signing authority and make sure everyone has their CA...and
they *should*...but realistically, that is probably relatively rare at
the moment, because it is not particularly easy to accomplish
(distributing the CA everywhere it needs to go is still a Hard Problem,
though it has gotten a lot better).

Given the reality of human nature, even when the documentation
accompanying the HTTPS initiative is good, there will *still* be someone
who hasn't followed the internal rules, yet you really need to talk to
the piece of infrastructure they are maintaining.  At least that one is
short term problem (for some definition of "short" that may be several
months long), but it does exist.

In addition, as has been mentioned before, self-signed certs are often
embedded in *devices* from vendors (I'm looking at you, Cisco).  This is
another area where security conciousness has gotten better (the cert
exists) but isn't good yet (the cert is self-signed and replacing it
isn't trivial when it is even possible; and, because the self-signed cert
happens by defaultit gets left in place).  And in the case of those
embedded certs, the cert can wind up *invalid* (expired) as well as
self-signed.  (This last item is where my concern about being able
to talk to invalid certs comes from.)

And yes, I have encountered all of this in the wild.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Mon, 01 Sep 2014 08:10:58 +1000, Nick Coghlan  wrote:
> On 1 Sep 2014 07:43, "Christian Heimes"  wrote:
> >
> > On 31.08.2014 08:09, Nick Coghlan wrote:
> > > As Antoine says here, I'm also opposed to adding more Python specific
> > > configuration options. However, I think there may be something
> > > worthwhile we can do that's closer to the way browsers work, and has
> > > the significant benefit of being implementable as a PyPI module first
> > > (more on that in a separate reply).
> >
> > I'm on your and Antoine's side and strictly against any additional
> > environment variables or command line arguments. That would make the
> > whole validation process even more complex and harder to understand.
> >
> > There might be a better option to give people and companies the option
> > to tune the SSL module to their needs. Python already have a
> > customization hook for the site module called sitecustomize. How about
> > another module named sslcustomize? Such a module could be used to tune
> > the ssl module to the needs of users, e.g. configure a different default
> > context, add certificates to a default context etc.
> >
> > Companies could install them in a system global directory on their
> > servers. Users could put them in their own user site directory and even
> > each virtual env can have one sslcustomize of its own. It's fully
> > backward compatible, doesn't add any flags and developers have the full
> > power of Python for configuration and customization.
> 
> And means a user specific store (if one became available) could be
> configured there.
> 
> Yes, I think this would address my concerns, especially if combined with a
> clear recipe in the documentation on how to optionally disable cert
> validation at the application layer.
> 
> Assuming sslcustomize was in site-packages rather than the standard library
> directories, you would also be able to use virtual environments with an
> appropriate sslcustomize module to disable cert checking even if the
> application you were running didn't support direct configuration.

It sounds like this would address my concerns as well (I don't really
care *how* it is implemented as long as I don't have to touch the
code of a third party application when I upgrade my python version to
3.5...remember, the context here is backward compatibility concerns).

Does it address the issue of accepting an invalid cert, though?

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR

[R. David Murray]

On Sun, 31 Aug 2014 20:14:50 -0700, Dan Stromberg  wrote:
> On Sun, Aug 31, 2014 at 3:28 PM, Greg Ewing  
> wrote:
> > Victor Stinner wrote:
> >>
> >> As written in the PEP, if you want to be notified of the signal, set a
> >> signal handler which raises an exception.
> >
> > I'm not convinced that this covers all possible use cases.
> > It might be all right if you have control over the signal
> > handler, but what if you don't?
> >
> > I think it's best if the functions in the os module remain
> > thin wrappers that expose the OS functionality as fully and
> > directly as possible. Anything else should be provided
> > by higher-level facilities.
> 
> I'm inclined to agree about keeping the os module thin.  If we were to
> recreate Python today, from scratch, it might make sense to hide this
> by default, but now there's almost certainly code out there that
> depends on the current behavior.
> 
> But I also agree that it's hard to pin down which higher level Python
> library calls are going to be using system calls.  My
> http://stromberg.dnsalias.org/~strombrg/pypty/ program had a problem
> with window resizing for a while (SIGWINCH), and although I use it
> pretty much daily now without problems, I'm still not sure I got 100%
> of the possibilities covered.
> 
> Fortunately, wrapping a system call can be as simple as:
> 
> def retry_on_eintr(function, *args, **kw):
> '''
> Retry a system call until one of the following happens:
> 1) It succeeds
> 2) It errors with something other than EINTR
> '''
> 
> while True:
> try:
> return function(*args, **kw)
> except OSError:
> nothing, extra, nothing2 = sys.exc_info()
> dummy = nothing
> dummy = nothing2
> if extra.errno == errno.EINTR:
> continue
> else:
> raise
> 
> Note that os.read() and os.write() need different handling.

Personally, I really want Python to handle EINTR for me.  And indeed,
that has been what we have been doing for a while now, piece by piece,
bug by bug.  Victor just wants to systematize and document that, and I
think that's a good idea.

We've been consistently treating lack of handling of EINTR as a bug.
If there are *real* cases where that causes a backward compatibility
problem, then we need to know.  But so far, we have gotten zero
complaints about the cases that we have fixed.

--David

PS: I recently switched from using selectors to using a timeout on a
socket because in that particular application I could, and because
reading a socket with a timeout handles EINTR (in recent python
versions), whereas reading a non-blocking socket doesn't.  Under the
hood, a socket with a timeout is a non-blocking socket.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR

[R. David Murray]

On Mon, 01 Sep 2014 08:30:27 +0300, Marko Rauhamaa  wrote:
> "R. David Murray" :
> 
> > PS: I recently switched from using selectors to using a timeout on a
> > socket because in that particular application I could, and because
> > reading a socket with a timeout handles EINTR (in recent python
> > versions), whereas reading a non-blocking socket doesn't. Under the
> > hood, a socket with a timeout is a non-blocking socket.
> 
> Under what circumstances would a nonblocking socket generate an EINTR?

Windows.  Enough said?

The exact error message was:

BlockingIOError on my non-blocking socket: 'a non-blocking socket
operation could not be completed immediately"

Needless to say, I was not expecting this, and was about to tear my
remaining hair out about having to completely restructure the code in
order to be able to handle an EINTR on a read on an FD that I got back
from select as ready, until I realized that the way the code had evolved
the only thing I still needed the select for was the timeout, and that
the EINTR bug in sockets with a timeout had already been fixed (thank
goodness I'm able to use python3.4 for this project).  I got lucky, but
this is clearly a serious problem for writing selectors based code on
Windows.

This should tell you just about everything you need to know about why we
want to fix this problem so that things work cross platform.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR

[R. David Murray]

On Mon, 01 Sep 2014 14:15:52 +0300, Marko Rauhamaa  wrote:
> Charles-François Natali :
> 
> >> Which raises an interesting question: what happens to the os.read()
> >> return value if SIGINT is received?
> >
> > There's no return value, a KeywordInterrupt exception is raised.
> > The PEP wouldn't change this behavior.
> 
> Slightly disconcerting... but I'm sure overriding SIGINT would cure
> that. You don't want to lose data if you want to continue running.
> 
> > As for the general behavior: all programming languages/platforms
> > handle EINTR transparently.
> 
> C doesn't. EINTR is there for a purpose. I sure hope Python won't bury
> it under opaque APIs.
> 
> The two requirements are:
> 
>  * Allow the application to react to signals immediately in the main
>flow.

You don't want to be writing your code in Python then.  In Python
you *never* get to react immediately to signals.  The interpreter
sets a flag and calls the python signal handler later.  Yes, the
call is ASAP, but ASAP is *not* "immediately".

>  * Don't lose information.
> 
> 
> Marko
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Tue, 02 Sep 2014 22:16:18 -, Alex Gaynor  wrote:
> This whole scenario seems to be predicated on a siutation where: You have a
> peer whose certificate you can't change, and you have a piece of code you 
> can't
> change, and you're going to upgrade your Python installation, and you want to
> talk to this peer, and you need to use an encrypted channel, but don't really
> care if it's being MITM'd. It doesn't seem to me that this is reasonably
> Python's responsibility to deal with the fact that you have no ability to
> upgrade any of your infrastructure, except your Python version.

I would say that is an excellent summary, except that you are sometimes
*forced* to use an encrypted channel (the device only opens the https port).
That is, in the real messy world of network devices, accepting an
invalid cert is not a nonsensical operation.  (Of course, what I really
want is to be able to say "I know this cert has invalid fields, accept
it anyway, but warn me if it changes when I connect again".)

It's a good point that the actual number of people who will be hit by
this and can't hack the code in question will be (possibly vanishingly)
small, especially if you only consider python3.  But we've already had
one call for a backport :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Tue, 02 Sep 2014 20:59:54 -0400, Terry Reedy  wrote:
> On 9/2/2014 7:47 PM, Glyph Lefkowitz wrote:
> >
> > On Sep 2, 2014, at 4:28 PM, Nick Coghlan  > > wrote:
> >
> >> On 3 Sep 2014 09:08, "David Reid"  >> > wrote:
> 
> >> > Clearly this change should be backported to Python2.
> >>
> >> Proposing to break backwards compatibility in a maintenance release (...)
> 
> For code that depends on a bug, fixing the bug breaks compatibility 
> between releases without and with the fix.  Nothing new.  We 
> occasionally don't backpart disruptive bugfixes because of that. We also 
> backport some security fixes to otherwise unmaintained versions.
> 
> > As we keep saying, this is not a break in backwards compatibility, it's
> > a bug fix.
> 
> Disagreement on whether changing the default for certificate validation 
> is a 'bugfix' or 'enhancement' (and in the colloquial or python tracker 
> sense) the seems like the crux of the disagreement here.  The case for 
> 'bug', at least in the colloquial meaning, is at least plausible. On the 
> other hand, the tracker meaning of 'behavior' issue (contrary to 
> intention and doc) is more strict in including implementation bugs but 
> not necessarily design bugs.

Up to this point the suggestion has been to change the default only in
3.5, and what we are arguing about is what kind of and how much aid to
give to those for whom 3.5 breaks backward compatibility.  So arguing
about feature versus bug is irrelevant.

The top proposal so far is an sslcustomize.py file that could be used to
either decrease or increase the default security.  This is a much less
handy solution than application options (eg, curl, wget) that allow
disabling security for "this cert" or "this CLI session".  It also is
more prone to unthinking abuse since it is persistent.  So perhaps
it is indeed not worth it.  (That's why I suggested an environment
variable...something you could specify on the command line for a one-off.)

The more I think about it the less I like sslcustomize.  I think I'd
rather do without the ability to turn off checking without modifying
code rather than introduce an intentional vector for a system-wide
decrease in security.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Wed, 03 Sep 2014 16:31:13 +0200, Antoine Pitrou  wrote:
> On Tue, 02 Sep 2014 21:29:16 -0400
> "R. David Murray"  wrote:
> > 
> > The top proposal so far is an sslcustomize.py file that could be used to
> > either decrease or increase the default security.  This is a much less
> > handy solution than application options (eg, curl, wget) that allow
> > disabling security for "this cert" or "this CLI session".  It also is
> > more prone to unthinking abuse since it is persistent.  So perhaps
> > it is indeed not worth it.  (That's why I suggested an environment
> > variable...something you could specify on the command line for a one-off.)
> 
> I'll be fine with not adding any hooks at all, and letting people
> configure their application code correctly :-)

Again, the problem arises when it is not *their* application code, but
a third party tool that hasn't been ported to 3.5.

I'm OK with letting go of this invalid-cert issue myself, given the lack
of negative feedback Twisted got.  I'll just keep my fingers crossed.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Wed, 03 Sep 2014 20:37:38 +0200, Antoine Pitrou  wrote:
> On Wed, 3 Sep 2014 10:54:55 -0700
> Guido van Rossum  wrote:
> > Today (working at Dropbox, a much smaller company!) I don't
> > even remember the last time I had to deal with such a browser
> > complaint -- internal services here all redirect to SSL, and not a
> > browser that can find fault with their certs.
> 
> Good for you. I still sometimes get warnings about expired certificates
> - and sometimes ones that don't exactly match the domain being
> fetched (for example, the certificate wouldn't be valid for that
> specific subdomain - note that CAs often charge a premium for multiple
> subdomains, which why small or non-profit Web sites sometimes skimp on
> them).
> 
> You shouldn't assume that the experience of well-connected people in
> the Silicon Valley is representative of what people over the world
> encounter. Yes, where there's a lot of money and a lot of accumulated
> domain competence, security procedures are updated and followed more
> scrupulously...

Heck, yesterday I got invalid certs from...I think it was roku.com, but
in any case not some obscure little company...the actual cert was an
akamai cert, which means something is configured wrong somewhere.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Wed, 03 Sep 2014 10:09:36 -0700, Ethan Furman  wrote:
> On 09/03/2014 08:58 AM, R. David Murray wrote:
> >
> > I'm OK with letting go of this invalid-cert issue myself, given the lack
> > of negative feedback Twisted got.  I'll just keep my fingers crossed.
> 
> I apologize if I missed this point, but if we have the source code then it is 
> possible to go in and directly modify the 
> application/utility to be able to talk over https to a router with an invalid 
> certificate?  This is an option when 
> creating the ssl_context?

The immediately preceding paragraph that you didn't quote said that the
context was 3rd party applications, not source code under your control.
Yes, you can (usually) still hack the source, but there are good reasons to
prefer to not do that, unfamiliarity with the codebase being just one of
them.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python docs about comparisons vs. CPython reality

[R. David Murray]

On Sat, 06 Sep 2014 13:34:37 +0200, Jan Kaliszewski  
wrote:
> Are they bugs in the Python docs or just some CPython implementation
> details that are purposely not documented? (but then, again, some of
> the docs seem to be at least not precise...):

You might want to read http://bugs.python.org/issue12067 and contribute
your thoughts there.  There are indeed documentation improvements
under discussion.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Sat, 13 Sep 2014 21:06:21 +1200, Nick Coghlan  wrote:
> On 13 Sep 2014 10:18, "Jeff Allen"  wrote:
> > 4. I think (with Antoine) if Jython supported PEP-383 byte smuggling, it
> would have to do it the same way as CPython, as it is visible. It's not
> impossible (I think), but is messy. Some are strongly against.
> 
> It may be worth trying *without* it (i.e. treat "surrogateescape" as
> equivalent to "strict" initially), and seeing how you go. The main purpose
> of surrogateescape in CPython 3 is to recreate the "arbitrary 8-bit data
> round trips work on POSIX" aspect of CPython 2, which doesn't apply in
> exactly the same way on Jython.
> 
> Compared to the 8-bit vs 16-bit str discrepancy that exists in Python 2,
> "surrogateescape is equivalent to strict" seems like a relatively small
> discrepancy in behaviour.

That would totally break the email package.

It would of course be possible to rewrite email to not use surrogate
escape, but it is a seriously non-trivial undertaking.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Tue, 16 Sep 2014 13:51:23 +1000, Chris Angelico  wrote:
> On Tue, Sep 16, 2014 at 1:34 PM, Stephen J. Turnbull  
> wrote:
> > Jim J. Jewett writes:
> >
> > > In terms of best-effort, it is reasonable to treat the smuggled bytes
> > > as representing a character outside of your unicode repertoire
> >
> > I have to disagree. If you ever end up passing them to something that
> > validates or tries to reencode them without surrogateescape, BOOM!
> > These things are the text equivalent of IEEE NaNs.  If all you know
> > (as in the stdlib) is that you have "generic text", the only fairly
> > safe things to do with them are (1) delete them, (2) substitute an
> > appropriate replacement character for them, (3) pass the text
> > containing them verbatim to other code, and (4) reencode them using
> > the same codec they were read with.
> 
> Don't forget, these are *errors*. These are bytes that cannot be
> correctly decoded. That's not something that has any meaning
> whatsoever in text; so by definition, the only things you can do are
> the four you list there (as long as "codec" means both the choice of
> encoding and the use of the surrogateescape flag). It's like dealing
> with control characters when you need to print something visually,
> except that they have an official solution [1] and surrogateescape is
> unofficial. They're not real text, so you have to deal with them
> somehow.

That isn't the case in the email package.  The smuggled bytes are not
errors[*], they are literally smuggled bytes.  But, as Stephen said, the
only things email does with them are the last three of the four he
listed (if you read (3) as passing it between parts of the email
package): the data comes in as text mixed with binary, and the email
package parses it until it knows what the binary is supposed to be,
turns it back into bytes, and decodes it properly.  The goal is to never
let the smuggled bytes escape out the email APIs as surrogateescape
encoded text; though, in practice, this being consenting-adults Python
and code not being bug free, there are places where people have used the
knowledge of how surrogateescape is used by email to work around both
API and code bugs.

--David

[*] Some of the encoded bytes *are* errors (non-ascii in headers or
undecodable bytes in whatever the CTE/charset is), and in that case
email may just turn them back into error bytes in the output, but only
*some* of the smuggled bytes are actually errors (and none are if the
message is RFC compliant).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Wed, 17 Sep 2014 01:27:44 +1000, Chris Angelico  wrote:
> On Wed, Sep 17, 2014 at 1:00 AM, R. David Murray  
> wrote:
> > That isn't the case in the email package.  The smuggled bytes are not
> > errors[*], they are literally smuggled bytes.
> 
> But they're not characters, which is what Stephen and I were saying -
> and contrary to what Jim said about treating them as characters. At
> best, they represent characters but in some encoding other than the
> one you're using, and you have no idea how many bytes form a character
> or anything. So you can't, for instance, word-wrap the text, because
> you can't know how wide these unknown bytes are, whether they
> represent spaces (wrap points), or newlines, or anything like that.
> You can't treat them as characters, so while you have them in your
> string, you can't treat it as a pure Unicode string - it''s a Unicode
> string with smuggled bytes.

Well, except that I do.  The email header parsing algorithms all work
fine if I treat the surrogate escaped bytes as 'unknown junk' and just
parse based on the valid unicode.  (Unless the header is so garbled that
it can't be parsed, of course, at which point it becomes an invalid
header).

You are right about the wrapping, though.  If a header with invalid
bytes (and in this scenario we *are* talking about errors) needs to
be wrapped, we have to first decode the smuggled bytes and turn it
into an 'unknown-8bit' encoded word before we can wrap the header.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Wed, 17 Sep 2014 04:02:11 +1000, Chris Angelico  wrote:
> On Wed, Sep 17, 2014 at 3:46 AM, R. David Murray  
> wrote:
> >> You can't treat them as characters, so while you have them in your
> >> string, you can't treat it as a pure Unicode string - it''s a Unicode
> >> string with smuggled bytes.
> >
> > Well, except that I do.  The email header parsing algorithms all work
> > fine if I treat the surrogate escaped bytes as 'unknown junk' and just
> > parse based on the valid unicode.  (Unless the header is so garbled that
> > it can't be parsed, of course, at which point it becomes an invalid
> > header).
> 
> Do what, exactly? As I understand you, you treat the unknown bytes as
> completely opaque, not representing any characters at all. Which is
> what I'm saying: those are not characters.

Yes.  I thought you were saying that one could not treat the string with
smuggled bytes as if it were a string.  (It's a string that can't be
encoded unless you use the surrogateescape error handler, but it is
still a string from Python's POV, which is the point of the error
handler).

Or, to put it another way, your implication was that there were no
string operations that could be usefully applied to a string containing
smuggled bytes, but that is not the case.  (I may well have read an
implication that was not there; if so I apologize and you can ignore the
rest of this :)  Basically, we are pretending that the each smuggled
byte is single character for string parsing purposes...but they don't
match any of our parsing constants.  They are all "any character" matches
in the regexes and what have you.  Of course, this only works in
contexts where we can ignore or "carry along" the smuggled bytes as
being components of "arbitrary text" portions of the syntax, and we must
take care to either replace them with valid unicode error glyphs or turn
the string of which the are a part into binary using the same codec and
error handler as we used to ingest them to begin with before emitting
them.  And, of course, we can't *modify* the sections containing the
smuggled bytes, only the syntax-matched sections that surround them; and
things like line wrapping are just an invitation to ugliness and bugs
even if you kept the smuggled bytes sections internally intact.

Finally, to explain what I meant by "except that I do": when I added
back binary support to the email package in Python3, initially I *did
not change the parsing algorithms* in the code.  I just smuggled the
bytes, and then dealt with the encoding/decoding at the API boundaries.
This is the same principle used when dealing with filenames in the API
of Python itself.  *Except* at that boundary, I do not need to worry
about whether a particular string contains smuggled bytes or not.[*]

> If you, instead, represented the header as a list with some str
> elements and some bytes, it would be just as valid (though much harder
> to work with); all your manipulations are done on the str parts, and
> the bytes just tag along for the ride.

Quite a bit harder, which is why I don't do that.

> > You are right about the wrapping, though.  If a header with invalid
> > bytes (and in this scenario we *are* talking about errors) needs to
> > be wrapped, we have to first decode the smuggled bytes and turn it
> > into an 'unknown-8bit' encoded word before we can wrap the header.
> 
> Yeah, and that's going to be a bit messy. If you get 60 characters
> followed by 30 unknown bytes, where do you wrap it? Dare you wrap in
> the middle of the smuggled section?

The point of RFC2047 encoded words is that they are an ASCII
representation of binary data, so once the bytes are "properly" Content
Transfer Encoded (as being in an unknown charset) the string contains no
smuggled bytes and can be wrapped.

--David

[*] I worried a lot that this was re-introducing the bytes/string
problem from python2.  The difference is that if the smuggled bytes
escape from the email API, that's a bug in the email package.  So user
code using the library is *not* in danger of getting mysterious encoding
errors when one day the input is international where before it was all
ASCII.  (Absent bugs in the library.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Wed, 17 Sep 2014 08:57:21 +0900, "Stephen J. Turnbull"  
wrote:
> As long as the Java string manipulation functions don't check for
> surrogates, you should be fine with this representation.  Of course I
> suppose your matching functions (etc) don't check for them either, so
> you will be somewhat vulnerable to bugs due to treating them as
> characters.  But the same is true for CPython, AFAIK.

>From my point of view, the string function laxness is a feature, not a
bug.  But I get what you mean.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Wed, 17 Sep 2014 14:42:56 +1000, Steven D'Aprano  wrote:
> On Wed, Sep 17, 2014 at 11:14:15AM +1000, Chris Angelico wrote:
> > On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray  
> > wrote:
> 
> > > Basically, we are pretending that the each smuggled
> > > byte is single character for string parsing purposes...but they don't
> > > match any of our parsing constants.  They are all "any character" matches
> > > in the regexes and what have you.
> > 
> > This is slightly iffy, as you can't be sure that one byte represents
> > one character, but as long as you don't much care about that, it's not
> > going to be an issue.
> 
> This discussion would probably be a lot more easy to follow, with fewer 
> miscommunications, if there were some examples. Here is my example, 
> perhaps someone can tell me if I'm understanding it correctly.
> 
> I want to send an email including the header line:
> 
> 'Subject: “NOBODY expects the Spanish Inquisition!”'
> 
> Note the curly quotes. I've read the manifesto "UTF-8 Everywhere" so I 
> do the right thing and encode it as UTF-8:
> 
> b'Subject: \xe2\x80\x9cNOBODY expects the Spanish Inquisition!\xe2\x80\x9d'

That won't work until email supports RFC 6532.  Until then, you can only
use ascii and encoded words successfully.  So just having the curly
quotes is a buggy enough program.

> but it's not up to Python's email package to throw those invalid bytes 
> out or permantly replace them with something else. Also, we want to work 
> with Unicode strings, not byte strings, so there has to be a way to 
> smuggle those three bytes into Unicode, without ending up with either 
> the replacement bytes:
> 
> # using the 'replace' error handler
> 'Subject: ���NOBODY expects the Spanish Inquisition!”'

What you'll get if you request a text copy of that header is

  'Subject: ���NOBODY expects the Spanish Inquisition!���'

> Am I right so far?
> 
> So the email package uses the surrogate-escape error handler and ends up 
> with this Unicode string:
> 
> 'Subject: \udc9c\udc80\udce2NOBODY expects the Spanish Inquisition!”'

Except that it encodes the closing quote, too :)

> which can be encoded back to the bytes we started with.

Right.  If you serialize the message as bytes, the bytes are recovered
and output when that header is output.

Now, once we support RFC 6532, you will be exactly right, as we will
then have the option of handling utf-8 encoded headers, and we will do
that using the utf-8 codec to ingest headers, and the surrogateescape
error handler to handle exactly the kind of bad data you postulate.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

Sorry for the mojibake.  I've not yet gotten around to actually using
the email package to write a smarter replacement for nmh, which is what
I use for email, and I always forget that I need to manually tell nmh
when there non-ascii in the message...

On Wed, 17 Sep 2014 03:02:33 -0400, "R. David Murray"  
wrote:
> On Wed, 17 Sep 2014 14:42:56 +1000, Steven D'Aprano  
> wrote:
> > On Wed, Sep 17, 2014 at 11:14:15AM +1000, Chris Angelico wrote:
> > > On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray  
> > > wrote:
> > 
> > > > Basically, we are pretending that the each smuggled
> > > > byte is single character for string parsing purposes...but they don't
> > > > match any of our parsing constants.  They are all "any character" 
> > > > matches
> > > > in the regexes and what have you.
> > > 
> > > This is slightly iffy, as you can't be sure that one byte represents
> > > one character, but as long as you don't much care about that, it's not
> > > going to be an issue.
> > 
> > This discussion would probably be a lot more easy to follow, with fewer 
> > miscommunications, if there were some examples. Here is my example, 
> > perhaps someone can tell me if I'm understanding it correctly.
> > 
> > I want to send an email including the header line:
> > 
> > 'Subject: “NOBODY expects the Spanish Inquisition!”'
> > 
> > Note the curly quotes. I've read the manifesto "UTF-8 Everywhere" so I 
> > do the right thing and encode it as UTF-8:
> > 
> > b'Subject: \xe2\x80\x9cNOBODY expects the Spanish Inquisition!\xe2\x80\x9d'
> 
> That won't work until email supports RFC 6532.  Until then, you can only
> use ascii and encoded words successfully.  So just having the curly
> quotes is a buggy enough program.
> 
> > but it's not up to Python's email package to throw those invalid bytes 
> > out or permantly replace them with something else. Also, we want to work 
> > with Unicode strings, not byte strings, so there has to be a way to 
> > smuggle those three bytes into Unicode, without ending up with either 
> > the replacement bytes:
> > 
> > # using the 'replace' error handler
> > 'Subject: ���NOBODY expects the Spanish Inquisition!”'
> 
> What you'll get if you request a text copy of that header is
> 
>   'Subject: ���NOBODY expects the Spanish Inquisition!���'
> 
> > Am I right so far?
> > 
> > So the email package uses the surrogate-escape error handler and ends up 
> > with this Unicode string:
> > 
> > 'Subject: \udc9c\udc80\udce2NOBODY expects the Spanish Inquisition!”'
> 
> Except that it encodes the closing quote, too :)
> 
> > which can be encoded back to the bytes we started with.
> 
> Right.  If you serialize the message as bytes, the bytes are recovered
> and output when that header is output.
> 
> Now, once we support RFC 6532, you will be exactly right, as we will
> then have the option of handling utf-8 encoded headers, and we will do
> that using the utf-8 codec to ingest headers, and the surrogateescape
> error handler to handle exactly the kind of bad data you postulate.
> 
> --David
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke

[R. David Murray]

On Fri, 19 Sep 2014 10:16:20 -0400, Barry Warsaw  wrote:
> The way I look at it is that "/usr/bin/python" is user interface.
> Distributions are completely free to choose whichever Python they want for
> system scripts, and it's great to see that Fedora is well on their way to
> making Python 3 the default for system scripts.  Debian is also making good
> progress, though we likely won't complete the transition until Jessie+1.

Yep, user interface.  Therefore I think the distribution should let the
user choose :)

Most users will chose python2, because that's what other scripts they
might download and install will expect.  So that should be the default.
But I'll choose python3 (and have, on most of my gentoo linux systems,
which supports having both quite well, for system installed packages).

On the gripping hand, given what Fedora is doing, it is not that hard to
change (or create) the symlinks as an end user to point to the python3
programs even if Fedora doesn't support it directly, so I don't see a
problem with the proposed strategy.

> If the user wants to invoke Python 3, it's not hard to type 'python3' and I
> think that's the message we should be spreading.  That already seems pretty
> ingrained in user habits afaict.

As Donald pointed out, that presents a problem for the future.  But
since there is no good alternative to putting /usr/bin/python3 in the
shebang of scripts that require python3, I don't think there is a
solution and we're just going to have to cross that further bridge when
we come to it.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sysadmin tasks

[R. David Murray]

On Wed, 01 Oct 2014 20:35:29 +1300, Shorya Raj  wrote:
> Just curious, is there any sort of tasklist for any sort of sysadmin sort
> of work surrounding CPython development? There seem to be plenty of tasks
> for the actual coding part, but it would be good to get something up for
> the more systems admin side of things. If there is no one managing that
> side yet, I would be more than happy to start to do so.

We have a sysadmin team (python-infrastructure is the mailing list,
there's also a #python-infra channel on freenode).  I'm sure they will
be happy for additional help!  I don't know that they have a formal task
list, but they might.  (Although I also do sysadmin stuff, I've confined
myself to bugs.python.org for my python sysadmin work...and that isn't
currently handled by the python-infrastructure team).

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] bytes-like objects

[R. David Murray]

Over the past while we've been cleaning up the docs in the area of "how
do we refer to bytes, bytearray, memoryview, etc, etc?" in the APIs that
deal with bytes.  As you may or may not remember, we settled on the term
'bytes-like object', and have changed the docs to (we hope) consistently
use this term, which has a glossary entry most of the references to it
link to.

I just committed (to 3.5) the final changes for issue 16518, the change
to consistently using the term 'bytes-like object' for things that
support the buffer interface.  This means that messages that previously
read:

'sometype' does not support the buffer interface

now read

a bytes-like object is required, not 'sometype'

The 'buffer interface' messages were rather confusing, since you have to
dig to find out what the 'buffer interface' is.  It isn't any sort of
python object, nor is there any object in python3 that has a name
related to it.  'bytes-like object', on the other hand, is fairly
intuitive[*], and if you look it up in the glossary it links to the
explanation of the buffer interface.

We felt it was worth explicitly mentioning this patch on python-dev to point
out to everyone that the docs and error messages now use a consistent
terminology, instead of the mis-mash we had before, which should make it easier
to help people with issues where this comes up.

If there are objections to this change to the messages it is easy enough to
back out, but personally I find the new error messages *much* clearer, and I'm
an experienced dev.

(This work was done by Ezio Melotti, but I committed the final patch as
part of my quest to clear the 'commit review' queue in the tracker.)

--David

[*] the more esoteric cases are not often encountered in regular (as opposed to
NumPy) code, and when they are, the extension to "something that can be
interpreted as a sequence of bytes" is pretty straightforward conceptually.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fixing 2.7.x

[R. David Murray]

On Mon, 06 Oct 2014 21:18:23 +0200, Christian Tismer  
wrote:
> On 06.10.14 20:55, Zachary Ware wrote:
> > On Mon, Oct 6, 2014 at 12:24 PM, Ned Deily  wrote:
> >> 3. security: "fixing issues exploitable by attackers such as crashes,
> >> privilege escalation and, optionally, other issues such as denial of
> >> service attacks. Any other changes are not considered a security risk
> >> and thus not backported to a security branch."
> >>= 3.2.x and 3.3.x
> >
> > 3.1 is still in this category, is it not?  According to PEP 375, it's
> > a few months past due for its last release.
> >
> > http://legacy.python.org/dev/peps/pep-0375/#maintenance-releases
> >
> 
> I don't think that the rules should be implicitly considered
> compatible between the 2.X and 3.X series.
> 
> Python 2.X has a history that extends to X==6, X==5 and
> even X==4, as a really conservative POV with an extent over more
> than 10 years.
> 
> I believe, such a thing does not exist for the Python 3.X series
> at all. My impression is that no 3.X user ever would want to stick
> with any older version.
> 
> Is that true, or am I totally wrong?

I don't think you are *totally* wrong, but I don't think you are really
right, either.  I myself have at least one system (I didn't check them
all) running 3.2 that I have intentionally only done security fixes on
rather than upgrade python3.  It's mostly laziness (given that my distro
provides the security updates), since I don't think there would be any
compatibility problems with the few python3 programs running on it, but
I'm likely not the only one.

So yes, the same rules should apply to python3 as apply to python2,
especially since more distros are about to start shipping python3 as
the system python (Arch has been since 2011).

3.1, however, is most likely a dead issue.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] mUTF-7 support?

[R. David Murray]

On Fri, 10 Oct 2014 01:33:58 +0200, Jesus Cea  wrote:
> On 10/10/14 01:08, Victor Stinner wrote:
> > When you say "IMAP4", do you mean any IMAP4 server? Do you have a list
> > of server vendors known to use the encoding mUTF-7?
> 
> All of them. IMAP4 protocol **REQUIRES** mUTF-7.

[...]

> I am volunteering and can even do the mercurial PUSH myself :-p. That is
> an advantage over some of those new encoding requests :-pp.
> 
> But then yes, I realize that this is a specialized tool (even if IMAP4
> is probably the most popular mail access protocol in the world), we can
> accommodate every use-case and there are tons of mUTF-7 libraries out
> there already.
> 
> So, I am asking :).

I see you are already nosy on issue 5305.  I don't think there is any
question that this is a useful and desired feature for python's imaplib.
If it makes sense to implement it as a codec, there is no reason *not*
to do that.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] mUTF-7 support?

[R. David Murray]

On Fri, 10 Oct 2014 04:28:21 +0200, Antoine Pitrou  wrote:
> On Thu, 9 Oct 2014 19:12:29 -0700
> Dan Stromberg  wrote:
> > On Thu, Oct 9, 2014 at 3:47 PM, Jesus Cea  wrote:
> > > I miss mUTF-7 support (as used to encode IMAP4 mailbox names) in Python,
> > > in the codecs module. As an european with a language with 27 different
> > > letters (instead of english 26), tildes, opening question marks, etc., I
> > > find it very inconvenient.
> > >
> > > This encoding is used basically only in IMAP4, I know. But IMAP4 is an
> > > important protocol and all projects related to it needs mUTF-7 support
> > > if they care about non-english alphabets. Everybody has already an
> > > implementation, waste of effort.
> > 
> > I've been parsing up a huge gmail account with no encoding errors,
> > using CPython 2.x and CPython 3.x.  I'd be surprised if there are no
> > foreign characters in any of the thousands of messages there - but
> > maybe I'm just being very lucky.  I'm not specifying a codec, and I
> > don't see a way of specifying one offhand.
> 
> AFAIU, this is specifically about mailbox names, not messages.

Specifically, it is about what we might better term mailbox
*folders*...that is, not what you would normally think of as the
'mailbox name', which is usually understood to be the thing before the @
in the email address (and can't contain non-ASCII yet...we need RFC 6855
support for that, and I'm not sure *anybody* has that yet).

In this context it is the names you give to folders on the IMAP
server...starting (usually) with INBOX and adding from there.  These
names are used in IMAP commands (ex: the 'select' or 'create' commands),
and IMAP uses mUTF-7 for those.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Internationalized email support (was mUTF-7 support?)

[R. David Murray]

On Fri, 10 Oct 2014 16:16:24 +0900, "Stephen J. Turnbull"  
wrote:
> R. David Murray writes:
> 
>  > in the email address (and can't contain non-ASCII yet...we need RFC 6855
>  > support for that, and I'm not sure *anybody* has that yet).
> 
> If it's an RFC, *somebody* has it *somewhere*. :-)

Ah, it looks like you may be right.  I didn't find any hits in the
first few pages on Google; the closest I got was:

https://groups.google.com/forum/?_escaped_fragment_=topic/mozilla.dev.apps.thunderbird/QBFibOEK5x4#!topic/mozilla.dev.apps.thunderbird/QBFibOEK5x4

I started writing a response that said "if so, it is hidden from
google".  Then I did some deeper digging, and found this:

http://googleblog.blogspot.com/2014/08/a-first-step-toward-more-global-email.html

So my search using the RFC number was just too narrow...since gmail is
IMAP based, Google itself must have *some* sort of RFC 6855 support.
(Unless they've done a hack job of this internationalization support
that only works for their own gmail clients, which given my past
experience working with the gmail IMAP API I would believe...although
I think the IMAP problems I found were mostly bugs rather than
intentional deviations from the standards).

We had a GSOC student (Milan Oberkirch) working on international email
support this past summer, and I believe you asked in the proposal review
phase why working on support for these RFCs was apropos.  At the time I
was kind of hoping Python could kickstart development of
internationalized email by providing tools to build experiments, but now
that Google has announced support, supporting it has become surprisingly
immediately relevant.  (The previously experimental postfix support for
RFC 6531 is targeted for official release in 2.12, which is another big
step forward.)

I'd better get to reviewing the rest of those patches sometime
soonthey include a preliminary patch for RFC 6855.  Unfortunately
the email package itself doesn't handle RFC 6532 email and there's no
proposed patch for that yet.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sad status of Python 3.x buildbots

[R. David Murray]

On Fri, 10 Oct 2014 18:00:06 +0200, Jesus Cea  wrote:
> On 10/10/14 17:56, Chris Angelico wrote:
> > Could I write a little
> > monitor at my end that asks every hour if my buildbots can be seen?
> 
> AFAIK maintainers already get an email if the buildbot vanishes long
> enough. I am more interested in getting an email when my buildbot is
> consistently red because somebody committed something it doesn't like
> two months ago...

I think they are supposed to, but I've never gotten one, not even when
my gentoo buildbots suffered a hardware failure.  A month or so later
Antoine emailed me, and I told him to remove them at least for now,
since I don't currently have replacement hardware.  (I'm hoping to fix
that, but I have to find the time...)

That said, there has never as far as I know been an active hook to email
the maintainer when the buildbot is red.  I don't know if buildbot
supports that or not, so that would be the first thing to check.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sat, 25 Oct 2014 05:45:24 -0700, "Tony Kelman"  wrote:
> As a developer of a (compiled) open-source library or application, wouldn't
> you love to be able to build binaries on Linux for Windows? With some work
> and build system patches, you can. For many projects it's a simple matter of
> ./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
> only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
> Arch. This is possible with a very, very long set of patches, many of which
> have been submitted by Roumen Petrov to the Python bug tracker - see
> http://bugs.python.org/issue17605 and other issues linked therein. Roumen
> has done a huge amount of work, and he and others who consider the MinGW-w64
> compiler important will continue to do so. (Thanks a million Roumen!)

Yes, I for one appreciate Roumen's work, even though I'm not currently
in a position to review the patches.

> > I could step in as maintainer for Cygwin and builds based on GCC using
> > mingw* APIs.
> >
> > Regards,
> > Roumen Petrov
> 
> A maintainer has volunteered. Others will help. Can any core developers
> please begin reviewing some of his patches? Even if just to say they need
> to be rebased. The lack of responses on the bug tracker is disheartening
> from an outside perspective. The pile of patches accumulating in external
> MinGW packaging projects is tantamount to a fork of CPython. It won't go
> away since there are dedicated packagers working to keep their MinGW-w64
> builds functional, even in the ad-hoc current state. The patches continue
> piling up, making it more difficult for everyone - except for the core
> Python developers if they continue to ignore the problem. Bring the people
> working on these patches into the fold as contributors. Review the patches.
> It would make Python as a language and a community even more diverse and
> welcoming.

IIUC, our general policy for bringing new platforms into core support,
as opposed to continuing to be a separately maintained "set of patches",
is that there be a "big enough" community of interest (I don't remember
the definition of "big enough") and that there be both committed
maintainers *and* at least one buildbot.

Being able to build windows packages on linux is a compelling argument,
but that only applies to building extensions, not the interpreter.

I would recommend starting with any patches that help MinGW that are not
MinGW specific but instead generally improve the build system and cross
compilation.  There have been a number of such issues opened and
improvements made beyond the MinGW related ones, some coming from
Debian, some coming from the Android community.

So  target those first.  My suggestion would be to pick a patch that is
believed to be commit ready, and come to #python-dev on freenode and
gently bug us until it gets committed.  Then pick the next one, and
repeat.  Working from simplest to more complex would also probably be a
good strategy :)

>From there I'd move on to patches that support using MinGW for building
extensions.  There will probably be useful to also get engaged with the
packaging folks.  And, at this point, we would NEED A BUILDBOT.  That
is, a machine that has whatever tools are required installed such that
tests added to the test suite to test MinGW support in distutils would
run, so we can be sure we don't break anything when making other
changes.

For compiling CPython itself with MinGW, we'd first need to develop a
consensus that we want to support it in core.  I'd say get building
extensions working first, and then make that argument.

By the time a bunch of patches get committed, we should be ready (read:
eager :) to promote someone to do it themselves.  Even if we never
decide to support compiling CPython itself with MinGW, I would hope that
getting it to work for extensions would greatly reduce the number of
patches needed to be maintained outside the tree in order to do so.  And
once at least one MinGW advocate is part of the core team, advocacy
becomes easier :)

--David

PS: one meta comment about the MinGW issues: I scanned a few from the
linked bug, and while the issues split the patches out, which is a great
start, there is no discussion on the issues for the individual patches
to give background and motivation and an explanation of what the patch
is trying to accomplish.  But you probably don't want to waste time on
improving ones that apply *only* to compiling CPython itself unless we
get consensus that we want to support that.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sat, 25 Oct 2014 21:10:23 +0100, Ray Donnelly  
wrote:
> On Sat, Oct 25, 2014 at 6:13 PM, Steve Dower  
> wrote:
> > (Apologies for the short reply, posting from my phone.)
> >
> > "MSVC can continue
> > to be the default compiler used for Python on Windows, none of Roumen's
> > patches change that. They would merely open up the choice for packagers and
> > users to build CPython (and extension modules, thanks to separate patches)
> > with alternate compilers, in cross-compilation or otherwise."
> >
> > Building CPython for Windows is not something that needs solving. The
> > culture on Windows is to redistribute binaries, not source, and both the
> > core team and a number of redistributors have this figured out (and it will
> > only become easier with VC14 and Python 3.5).
> 
> This is the second time you've used the vacuous "culture on Windows"
> argument, now with an added appeal to (vague) authority. That may be
> your opinion and that of some others, but there's a large number of
> people who don't care for using non-Free tools. IMHO building CPython
> on Windows using Open Source toolchains is very much something that
> needs merging upstream and supporting by default. What is it that you
> are afraid of if CPython can be compiled out of the box using
> mingw/MinGW-w64? Why are you fighting so hard against having option.
> If CPython wants to truly call itself an Open Source project then I
> consider being able to compile and cross-compile it with capable Open
> Source toolchains on all major platforms a requirement.

You are doing yourself a disservice by this last statement.  There
really can't be any question that Python is an open source project,
so insinuating that the CPython community is "doing something wrong"
is not going to win you friends and helpers.

A better approach would be to acknowledge that what we are currently
doing works well for supporting Windows (especially since we actually
have some engagement from Microsoft that is *getting some problems
fixed* in ways that help make things more open).

And then say, "wouldn't it be *really cool* if we could also build
CPython using an open source toolchain on Windows out of the box?".  You
might not get instant agreement on that (well, clearly you won't), but
it'd be much more likely you'd start garnering support.

Assume that people are well intentioned, and convince them your
suggestions will make things *even better* using positive arguments.
You might not succeed, but you'll have a much better chance.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sun, 26 Oct 2014 00:19:44 +0200, Antoine Pitrou  wrote:
> On Sun, 26 Oct 2014 09:06:36 +1100
> Chris Angelico  wrote:
> > On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou  wrote:
> > > How do you know this isn't a problem, since you haven't *tested* with
> > > MSVC?
> > > Why on Earth would you want to test your PEP work with an unsupported
> > > Windows compiler and runtime, rather than with the officially supported
> > > compiler and runtime?
> > 
> > This discussion revolved around supporting MinGW in addition to MSVC.
> > If it had been supported when I was doing that, I could have spun
> > myself up a Windows build and tested it.
> 
> My point is that your "Windows build" would not have the same behaviour
> as a MSVC-produced Windows build, and so testing it with it would not
> certify that your code would actually be compatible with genuine
> MSVC builds of CPython, which we will not stop supporting.

While true, I don't think that matters for Chris' point.  Given only the
ability to build with the MSVC toolchain, his code (which might even be
pure python for the purposes of this discussion) would not get tested on
Windows until committed and run by the buildbots.  If he could build
CPython using MinGW, he would, and would test his code on Windows.  Even
if there are C components and MSVC/MinGW compatibility issues are
revealed when the buildbots eventually run the code, still the number of
bugs present would probably be lower if he had tested it on Windows
than if he hadn't.

I know I for one do not generally test patches on Windows because I
haven't taken the time to learn how to build CPython on it.  Sure, I
could test pure python changes by applying patches to an installed
Python, but that's an ongoing pain and I'd rather learn to build CPython
on Windows and get to use the normal hg tools.

If I could use a more linux-like toolchain to build CPython on windows,
I would doubtless do much more testing on windows for stuff where I
think windows might behave differently (and I might look at more Windows
bugs...though frankly there are plenty of bugs for me to look at without
looking at Windows bugs).

This is not necessarily a compelling argument for MinGW support.
However, it *is* a valid argument, IMO.

Note: it can be made even less compelling by making it a lot easier to
build CPython on Windows without having an MSVC license (which I think
means not using the GUI, for which I say *yay* :).  I think Zach Ware
has been working on improving the Windows build process, and I keep
meaning to give it a try...

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sun, 26 Oct 2014 06:12:45 -0700, "Tony Kelman"  wrote:
> Steve Dower:
> > Building CPython for Windows is not something that needs solving.
> 
> Not in your opinion, but numerous packagers of MinGW-based native or
> cross-compiled package sets would love to include Python. The fact
> that they currently can't, without many patches, is a problem.

If this includes (or would likely include) a significant portion of the
Scientific Computing community, I would think that would be a compelling
use case.  We'd need to be educated more about the reasons why this
approach works better than remaining compatible with MSVC CPython so we
could evaluate the risks and reward intelligently.  (I wonder..."things
are going to fragment anyway if you (cpython) don't do anything" might
be an argument, if true...but wouldn't make the consequences any
easier to deal with :(

But as has been discussed, it seems better to focus first on fixing the
issues on which we are all in agreement (building extensions with MinGW).

> R. David Murray:
> > And, at this point, we would NEED A BUILDBOT.  That is, a machine that
> > has whatever tools are required installed such that tests added to the
> > test suite to test MinGW support in distutils would run, so we can be
> > sure we don't break anything when making other changes.
> 
> That's not too hard. I've done this for other projects. AppVeyor works if
> your build is short enough, and I've done cross-compilation from Travis
> CI for other projects. Or Jenkins, or a Vagrant VM. I don't know PSF's
> infrastructure, but I can offer guidance if it would help.

When I say "we need a buildbot", what I mean is that we need someone
willing to donate the resources and the *time and expertise* to setting
up and maintaining something that integrates with our existing buildbot
setup.  You set up a buildbot slave, request an id and password from
Antoine, keep the thing running, and respond in a timely fashion to
requests for help resolving issues that arise on the buildbot (both
buildbot issues and help-me-diagnose-this-failure issues).  After the
initial setup the load isn't generally heavy (I haven't had to do
anything with the OSX buildbot running on the machine in my dinning room
for months and months now, for example).

So your guidance would have to go to someone who was volunteering to
take on this task...there isn't anyone on the existing core team who
would have time to do it (if I'm wrong, I'm sure someone will speak up).
On the other hand, you don't have to be a committer to run a buildbot,
and there *are* people on the core-mentorship list who have expressed
interest in helping out with our automated testing infrastructure,
including (if I understand correctly) adding some level of integration
to other CI systems (which might just be messages to the IRC
channel)[*].  So that could be a fruitful avenue to explore.

--David

[*] This is an area in which I have an interest, but it hasn't gotten
high enough on my todo list yet for me to figure out exactly what the
current state of things is so I can help it along.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Wed, 29 Oct 2014 10:22:14 -0400, Tres Seaver  wrote:
> On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
> 
> > most developers on Windows do have access to Microsoft tool
> 
> I assume you mean python-dev folks who work on Windows:  it is certainly
> not true for the vast majority of develoeprs who use Python on Windows,
> who don't have the toolchain build their own C extensions.

I'm pretty sure he meant "most people who develop software for Windows",
even though that's not how he phrased it.  But this does not include, as
you point out, people who develop Python software that *also* works on
Windows.

If you are writing code targeted for Windows, I think you are very
likely to have an MSDN subscription of some sort if your package includes C
code.  I'm sure it's not 100%, though.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Thu, 30 Oct 2014 01:09:45 +1000, Nick Coghlan  wrote:
> (Paul Moore already covered most of this, but I'll go into a bit more
> detail in a couple of areas)
> 
> On 29 October 2014 00:46, Tony Kelman  wrote:
> > Stephen J. Turnbull:
> >> It should be evident by now that our belief is that the large majority
> >> of Windows users is well-served by the current model
> >
> >
> > This is not the case at all in the scientific community. NumPy and SciPy
> > put in a lot of extra work to come up with something that is compatible
> > with the MSVC build of CPython because they have to, not because they're
> > "happy to" jump through the necessary hoops.
> 
> Lots of folks are happy with POSIX emulation layers on Windows, as
> they're OK with "basically works" rather than "works like any other
> native application". "Basically works" isn't sufficient for many
> Python-on-Windows use cases though, so the core ABI is a platform
> native one, rather than a POSIX emulation.

Since some of the context here is scientific use of Python, it might be
a useful bit of perspective to know that, while there are doubtless many
scientists using windows and using the windows native interfaces
happily, the Software Carpentry bootcamps that aim to give scientists
the basic framework for making better use of computers and software and
programming have as one foundation the bash shell, taught on Windows via
git-bash.  That is, the common toolset being taught to scientists (by
Software Carpentry) is the posix one, even on Windows.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Wed, 29 Oct 2014 23:33:06 -0400, Terry Reedy  wrote:
> On 10/29/2014 4:05 PM, Paul Moore wrote:
> > On 29 October 2014 15:31, Nathaniel Smith  wrote:
> >>> You can use Express editions of Visual Studio.
> >>
> >> IIUC, the express edition compilers are 32-bit only, and what you actually
> >> want are the "SDK compilers":
> >> https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
> >>
> >> These are freely downloadable by anyone, no msdn subscription required, but
> >> only if you know where to find them!
> >>
> >> AFAICT the main obstacle to using MSVC to build python extensions (assuming
> >> it can handle your code at all) is not anything technical, but rather that
> >> there's no clear and correct tutorial on how to do it, and lots of 
> >> confusion
> >> and misinformation circulating.
> >
> > Would it help if I wrote a document explaining how to set up the MS
> > compilers (free and paid for) to allow building of Python extensions?
> 
> > There are a few provisos:
> >
> > 1. A lot of it will be pretty trivial: "If you have Vistal Studio
> > (full edition), install it. Done."
> > 2. It will be out of date very fast as the situation for Python 3.5+
> > will be trivial across the board.
> > 3. I don't have anywhere particularly authoritative to host it (maybe
> > the Python Wiki?) and it could easily get lost in the huge swamp of
> > variously outdated, over-complicated, or otherwise alternative
> > documents available. Ideally I'd like someone to suggest an "official"
> > location I could use.
> 
> There is already a section in the devguide for building Python itself, 
> with mostly the same info, except it may not be up to date.
> 
> > I don't want to do this if it won't be useful, as it'll take me a bit
> > of effort to confirm the process for the only non-trivial scenario
> > (64-bit Python 3.3/3.4 with free tools). But if people think it would
> > help, that's fine, I volunteer.
> 
> The devguide needs to be kept up to date.  If you open a tracker issue, 
> put me as nosy to review and commit.

The devguide is about building python itself.  Paul is talking about
building extensions.  That should go in the packaging docs.  I don't
*think* Paul is volunteering to explain how to build python itself,
that's pretty much our job :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The role of NotImplemented: What is it for and when should it be used?

[R. David Murray]

See issue 22766 for some background on this question.

On Mon, 03 Nov 2014 02:30:53 -0800, Ethan Furman  wrote:
> Just to be clear, this is about NotImplemented, not NotImplementedError.
> 
> tl;dr  When a binary operation fails, should an exception be raised or 
> NotImplemented returned?
> 
> 
> When a binary operation in Python is attempted, there are two possibilities:
> 
>- it can work
>- it can't work
> 
> The main reason [1] that it can't work is that the two operands are of 
> different types, and the first type does not know 
> how to deal with the second type.
> 
> The question then becomes: how does the first type tell Python that it cannot 
> perform the requested operation?  The most 
> obvious answer is to raise an exception, and TypeError is a good candidate.  
> The problem with the exception raising 
> approach is that once an exception is raised, Python doesn't try anything 
> else to make the operation work.
> 
> What's wrong with that?  Well, the second type might know how to perform the 
> operation, and in fact that is why we have 
> the reflected special methods, such as __radd__ and __rmod__ -- but if the 
> first type raises an exception the __rxxx__ 
> methods will not be tried.
> 
> Okay, how can the first type tell Python that it cannot do what is requested, 
> but to go ahead and check with the second 
> type to see if it does?  That is where NotImplemented comes in -- if a 
> special method (and only a special method) 
> returns NotImplemented then Python will check to see if there is anything 
> else it can do to make the operation succeed; 
> if all attempts return NotImplemented, then Python itself will raise an 
> appropriate exception [2].
> 
> In an effort to see how often NotImplemented is currently being returned I 
> crafted a test script [3] to test the types 
> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, deque, 
> and OrderedDict with the operations for 
> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, 
> __ilshift__, __imod__, __imul__, __ior__, __ipow__, 
> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, __mul__, 
> __or__, __pow__, __rshift__, __sub__, 
> __truediv__, and __xor__.
> 
> Here are the results of the 275 tests:
> 
> testing control...
> 
> ipow -- Exception  'subtype'> raised
> errors in Control -- misunderstanding or bug?
> 
> testing types against a foreign class
> 
> iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute 
> 'items'> raised instead of TypeError
> iand(Counter()) -- NotImplemented not returned, TypeError not raised
> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute 
> 'items'> raised instead of TypeError
> isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute 
> 'items'> raised instead of TypeError
> 
> 
> testing types against a subclass
> 
> mod(str()) -- NotImplemented not returned, TypeError not raised
> 
> iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'> 
> raised (should have worked)
> iand(Counter()) -- NotImplemented not returned, TypeError not raised
> ior(Counter()) -- Exception <'subtype' object has no attribute 'items'> 
> raised (should have worked)
> isub(Counter()) -- Exception <'subtype' object has no attribute 'items'> 
> raised (should have worked)
> 
> 
> Two observations:
> 
>- __ipow__ doesn't seem to behave properly in the 3.x line (that error 
> doesn't show up when testing against 2.7)
> 
>- Counter should be returning NotImplemented instead of raising an 
> AttributeError, for three reasons [4]:
>  - a TypeError is more appropriate
>  - subclasses /cannot/ work with the current implementation
>  - __iand__ is currently a silent failure if the Counter is empty, and 
> the other operand should trigger a failure
> 
> Back to the main point...
> 
> So, if my understanding is correct:
> 
>- NotImplemented is used to signal Python that the requested operation 
> could not be performed
>- it should be used by the binary special methods to signal type mismatch 
> failure, so any subclass gets a chance to work.
> 
> Is my understanding correct?  Is this already in the docs somewhere, and I 
> just missed it?
> 
> --
> ~Ethan~
> 
> [1] at least, it's the main reason in my code
> [2] usually a TypeError, stating either that the operation is not supported, 
> or the types are unorderable
> [3] test script at the end
> [4] https://bugs.python.org/issue22766 [returning NotImplemented was rejected]
> 
> -- 8< 
> 
> from collections import Counter, defaultdict, deque, OrderedDict
> from fractions import Fraction
> from decimal import Decimal
> from enum import Enum
> import operator
> import sys
> 
> py_ver = sys.ve

Re: [Python-Dev] The role of NotImplemented: What is it for and when should it be used?

[R. David Murray]

On Mon, 03 Nov 2014 15:05:31 +, Brett Cannon  wrote:
> On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman  wrote:
> 
> > Just to be clear, this is about NotImplemented, not NotImplementedError.
> >
> > tl;dr  When a binary operation fails, should an exception be raised or
> > NotImplemented returned?
> >
> 
> The docs for NotImplemented suggest it's only for rich comparison methods
> and not all binary operators:
> https://docs.python.org/3/library/constants.html#NotImplemented . But then
> had I not read that I would have said all binary operator methods should
> return NotImplemented when the types are incompatible.

Ethan opened an issue and then changed those docs, but I now believe
that the docs should be changed back (see the discussion in issue
22766).

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Real-world use of Counter

[R. David Murray]

On Wed, 05 Nov 2014 11:13:37 -0800, Ethan Furman  wrote:
> On 11/05/2014 10:56 AM, Raymond Hettinger wrote:
> > The in-place operations on counters are duck-typed.  They are intended (by 
> > design) to work with ANY type that has an
> > items() method.   The exception raised if doesn't have on is an 
> > AttributeError saying that the operand needs to have an
> > items() method.
> 
> It would still be duck-typed with a `hasattr` call on the second operand 
> checking for the necessary method, a TypeError 
> could just as easily state the problem is a missing `items()` method, and 
> then those three [*] in-place methods would 
> raise the same error as the 20(?) other Counter methods under similar 
> conditions.

The technique of duck typing as used in Python is you just let errors
bubble up when the input doesn't work.  Adding special case checks
should in general only be done if the resulting error message is
otherwise really confusing, which is not the case here.

As I said on the issue, there is no reason I can see to add extra code
just to turn an AttributeError into a TypeError.  The AttributeError
works just fine in letting you know your input type didn't work.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problem displaying the warning symbol

[R. David Murray]

On Sun, 16 Nov 2014 11:23:41 +0100, Stefano Borini 
 wrote:
> I report a finding (bug?) about the warning unicode symbol, as reported here
> 
> http://stackoverflow.com/questions/26919799/cannot-make-warning-sign-visible-on-osx-terminal

Bugs should be reported to bugs.python.org.  They just get lost here.

I'm doubtful this is a Python problem though.  More likely it is an
issue with the curses library on OSX.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[R. David Murray]

On Sun, 23 Nov 2014 00:59:42 +1000, Nick Coghlan  wrote:
> OK, different question. Has anyone here actually even *read* the workflow
> PEPs I wrote? They were on the agenda for the language summit, but got
> bumped due to lack of time (which I'm still annoyed about, given the
> comparatively inconsequential things that chewed up a whole lot of the day).

I have (but I'm core).  It's been a while, though, I need to review it.

> I've only had a couple of folks from outside the core dev community express
> interest in them. Personally, the lack of online editing support annoys me
> immensely whenever I need to work on PEPs or the devguide. I also think

Eh, I hate online editing ;).  But I understand its utility.

[...]

> My preferred answer remains setting up a srlf-hosted forge.python.org, but
> I've seen little evidence we have the capacity to deploy & maintain such a
> service effectively, given the relative lack of interest shown in the idea
> by almost everyone I've spoken to about it. Any progress has only come with
> a lot of pushing from me, and I just don't have the personal bandwidth to
> sustain that at this point. That's why the related PEPs were deferred, and
> the only responses I've received regarding potentially taking them over
> have come from folks outside the core development community, which really
> doesn't help very much in removing my availability as a bottleneck in the
> workflow improvement process.
> 
> If nobody wants to maintain a self-hosted forge, or even enable the folks
> that have expressed interest in setting it up & maintaining it, then the
> right answer is "don't do it" - we should use a commercial service instead.

I have an interest in this, but like you lack bandwidth.  I have no
employer that I could petition for 50% open source time, so any time I
spend on it is time I'm not billing...so my available time comes and
goes depending on my client situation, and I'm also one of the few
putting any time in on tracker maintenance (not that I have much lately,
Ezio has done most of it) and I've still got the email docs to finish up
as well as a list of bugs...yeah, bandwidth is a problem.

But maybe I can take over coordination of the volunteers that want to
work on the forge.  We've already got the mailing list set up.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My thinking about the development process

[R. David Murray]

On Fri, 05 Dec 2014 15:17:35 -0700, Eric Snow  
wrote:
> On Fri, Dec 5, 2014 at 1:04 PM, Brett Cannon  wrote:
> > We don't exactly have a ton of people
> > constantly going "I'm so bored because everything for Python's development
> > infrastructure gets sorted so quickly!" A perfect example is that R. David
> > Murray came up with a nice update for our workflow after PyCon but then ran
> > out of time after mostly defining it and nothing ever became of it (maybe we
> > can rectify that at PyCon?). Eric Snow has pointed out how he has written
> > similar code for pulling PRs from I think GitHub to another code review
> > tool, but that doesn't magically make it work in our infrastructure or get
> > someone to write it and help maintain it (no offense, Eric).
> 
> None taken.  I was thinking the same thing when I wrote that. :)
> 
> >
> > IOW our infrastructure can do anything, but it can't run on hopes and
> > dreams. Commitments from many people to making this happen by a certain
> > deadline will be needed so as to not allow it to drag on forever. People
> > would also have to commit to continued maintenance to make this viable
> > long-term.

The biggest blocker to my actually working the proposal I made was that
people wanted to see it in action first, which means I needed to spin up
a test instance of the tracker and do the work there.  That barrier to
getting started was enough to keep me from getting started...even though
the barrier isn't *that* high (I've done it before, and it is easier now
than it was when I first did it), it is still a *lot* higher than
checking out CPython and working on a patch.

That's probably the biggest issue with *anyone* contributing to tracker
maintenance, and if we could solve that, I think we could get more
people interested in helping maintain it.  We need the equivalent of
dev-in-a-box for setting up for testing proposed changes to
bugs.python.org, but including some standard way to get it deployed so
others can look at a live system running the change in order to review
the patch.

Maybe our infrastructure folks will have a thought or two about this?
I'm willing to put some work into this if we can figure out what
direction to head in.  It could well be tied in to moving
bugs.python.org in with the rest of our infrastructure, something I know
Donald has been noodling with off and on; and I'm willing to help with
that as well.

It sounds like being able to propose and test changes to our Roundup
instance (and test other services talking to Roundup, before deploying
them for real) is going to be critical to improving our workflow no
matter what other decisions are made, so we need to make it easier to
do.

In other words, it seems like the key to improving the productivity of
our CPython patch workflow is to improve the productivity of the patch
workflow for our key workflow resource, bugs.python.org.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker test instances (was: My thinking about the development process)

[R. David Murray]

On Sat, 06 Dec 2014 15:21:46 +, Brett Cannon  wrote:
> On Sat Dec 06 2014 at 10:07:50 AM Donald Stufft  wrote:
> > On Dec 6, 2014, at 9:11 AM, Brett Cannon  wrote:
> >
> >> On Fri Dec 05 2014 at 8:31:27 PM R. David Murray 
> >> wrote:
> >>> That's probably the biggest issue with *anyone* contributing to tracker
> >>> maintenance, and if we could solve that, I think we could get more
> >>> people interested in helping maintain it.  We need the equivalent of
> >>> dev-in-a-box for setting up for testing proposed changes to
> >>> bugs.python.org, but including some standard way to get it deployed so
> >>> others can look at a live system running the change in order to review
> >>> the patch.
> >>
> >> Maybe it's just me and all the Docker/Rocket hoopla that's occurred over
> >> the past week, but this just screams "container" to me which would make
> >> getting a test instance set up dead simple.
> >
> > Heh, one of my thoughts on deploying the bug tracker into production was
> > via a container, especially since we have multiple instances of it. I got
> > side tracked on getting the rest of the infrastructure readier for a web
> > application and some improvements there as well as getting a big postgresql
> > database cluster set up (2x 15GB RAM servers running in Primary/Replica
> > mode). The downside of course to this is that afaik Docker is a lot harder
> > to use on Windows and to some degree OS X than linux. However if the
> > tracker could be deployed as a docker image that would make the
> > infrastructure side a ton easier. I also have control over the python/
> > organization on Docker Hub too for whatever uses we have for it.
> >
> 
> I think it's something worth thinking about, but like you I don't know if
> the containers work on OS X or Windows (I don't work with containers
> personally).

(Had to fix the quoting there, somebody's email program got it wrong.)

For the tracker, being unable to run a test instance on Windows would
likely not be a severe limitation.  Given how few Windows people we get
making contributions to CPython, I'd really rather encourage them to
work there, rather than on the tracker.  OS/X is a bit more problematic,
but it sounds like it is also a bit more doable.

On the other hand, what's the overhead on setting up to use Docker?  If
that task is non-trivial, we're back to having a higher barrier to
entry than running a dev-in-a-box script...

Note also in thinking about setting up a test tracker instance we have
an additional concern: it requires postgres, and needs either a copy of
the full data set (which includes account data/passwords which would
need to be creatively sanitized) or a fairly large test data set.  I'd
prefer a sanitized copy of the real data.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My thinking about the development process

[R. David Murray]

On Mon, 08 Dec 2014 12:27:23 -0800, "Jim J. Jewett"  
wrote:
> Brett Cannon wrote:
> > 4. Contributor creates account on bugs.python.org and signs the
> >   [contributor agreement](https://www.python.org/psf/contrib/contrib-form/)
> 
> Is there an expiration on such forms?  If there doesn't need to be
> (and one form is good for multiple tickets), is there an objection
> (besides "not done yet") to making "signed the form" part of the bug
> reporter account, and required to submit to the CI process?  (An "I
> can't sign yet, bug me later" option would allow the current workflow
> without the "this isn't technically a patch" workaround for "small enough"
> patches from those with slow-moving employers.)

No expiration.  Whether or not we have a CLA from a given tracker id
is recorded in the tracker.  People also get reminded to submit a CLA
if they haven't yet but have submitted a patch.

> > At best core developers tell a contributor "please send your PR
> > against 3.4", push-button merge it, update a local clone, merge from
> > 3.4 to default, do the usual stuff, commit, and then push;
> 
> Is it common for a patch that should apply to multiple branches to fail
> on some but not all of them?

Currently?  Yes when 2.7 is involved.  If we fix NEWS, then it won't
be *common* for maint->default, but it will happen.

> In other words, is there any reason beyond "not done yet" that submitting
> a patch (or pull request) shouldn't automatically create a patch per
> branch, with pushbuttons to test/reject/commit?

Not Done Yet (by any of the tools we know about) is the only reason I'm
aware of.

> > Our code review tool is a fork that probably should be
> > replaced as only Martin von Loewis can maintain it.
> 
> Only he knows the innards, or only he is authorized, or only he knows
> where the code currently is/how to deploy an update?

Only he knows the innards.  (Although Ezio has made at least one patch
to it.)  I think Guido's point was that we (the community) shouldn't be
maintaining this private fork of a project that has moved on well beyond
us; instead we should be using an active project and leveraging
its community with our own contributions (like we do with Roundup).

> I know that there were times in the (not-so-recent) past when I had
> time and willingness to help with some part of the infrastructure, but
> didn't know where the code was, and didn't feel right making a blind
> offer.

Yeah, that's something that's been getting better lately (thanks,
infrastructure team), but where to get the info is still not as clear as
would be optimal.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition

[R. David Murray]

On Sat, 13 Dec 2014 10:17:59 -0500, Barry Warsaw  wrote:
> On Dec 13, 2014, at 12:29 AM, Donald Stufft wrote:
> 
> >For what it’s worth, I almost exclusively write 2/3 compatible code (and
> >that’s with the “easy” subset of 2.6+ and either 3.2+ or 3.3+) and 
> >doing so
> >does make the language far less fun for me than when I was writing 2.x only
> >code.
> 
> For myself, the way I'd put it is:
> 
> With the libraries I maintain, I generally write Python 2/3 compatible code,
> targeting Python 2.7 and 3.4, with 2.6, 3.3, and 3.2 support as bonuses,
> although I will not contort too much to support those older versions.  Doing
> so does make the language far less fun for me than when I am writing 3.x only
> code.  All applications I write in pure Python 3, targeting Python 3.4, unless
> my dependencies are not all available in Python 3, or I haven't yet had the
> cycles/resources to port to Python 3.  Writing and maintaining applications in
> Python 2 is far less fun than doing so in Python 3.

I think this is an important distinction.  The considerations are very
different for library maintainers than they are for application
maintainers.  Most of my work is in (customer) applications, and except
for one customer who insists on using an old version of RedHat, I've
been on "latest" python3 for those for quite a while now.  I suspect we
hear less here from people in that situation than would be proportional
to their absolute numbers.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition

[R. David Murray]

On Tue, 16 Dec 2014 10:48:07 -0800, Mark Roberts  wrote:
> On Tue, Dec 16, 2014 at 2:45 AM, Antoine Pitrou  wrote:
> >
> > Iterating accross a dictionary doesn't need compatibility shims. It's
> > dead simple in all Python versions:
> >
> > $ python2
> > Python 2.7.8 (default, Oct 20 2014, 15:05:19)
> > [GCC 4.9.1] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> d = {'a': 1}
> > >>> for k in d: print(k)
> > ...
> > a
> >
> > $ python3
> > Python 3.4.2 (default, Oct  8 2014, 13:08:17)
> > [GCC 4.9.1] on linux
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> d = {'a': 1}
> > >>> for k in d: print(k)
> > ...
> > a
> >
> > Besides, using iteritems() and friends is generally a premature
> > optimization, unless you know you'll have very large containers.
> > Creating a list is cheap.
> >
> 
> It seems to me that every time I hear this, the author is basically
> admitting that Python is a toy language not meant for "serious computing"
> (where serious is defined in extremely modest terms). The advice is also
> very contradictory to literally every talk on performant Python that I've
> seen at PyCon or PyData or ... well, anywhere. And really, doesn't it
> strike you as incredibly presumptuous to call the *DEFAULT BEHAVIOR* of
> Python 3 a "premature optimization"?

No.  A premature optimization is one that is made before doing any
performance analysis, so language features are irrelevant to that
labeling.  This doesn't mean you shouldn't use "better" idioms when they
are clear.  But if you are complicating your code because of performance
concerns *without measuring it* you are doing premature optimization, by
definition[*].

> Isn't the whole reason that the
> default behavior switch was made is because creating lists willy nilly all
> over the place really *ISN'T* cheap? This isn't the first time someone has

No.  In Python3 we made the iterator protocol more central to the
language.  Any performance benefit is actually a side effect of that
change.  One that was considered, yes, but in the context of the
*language* as a whole and not any individual program's performance
profile.  And "this doesn't make things worse for real world programs as
far as we can measure" is a more important criterion for this kind of
language change than "lets do this because we've measured and it makes
things better".

--David

[*] And yes, *we all do this*.  Sometimes doing it doesn't cost much.
Sometimes it does.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] contributing to multiprocessing

[R. David Murray]

On Thu, 08 Jan 2015 17:08:07 -0800, Ethan Furman  wrote:
> On 01/08/2015 03:21 PM, Davin Potts wrote:
> > 
> > I am interested in making some serious ongoing contributions around 
> > multiprocessing.
> 
> Great!
> 
> > Rather than me simply walking through that backlog, offering comments or 
> > encouragement here and there on issues, it
> > makes more sense for me to ask:  what is the right way for me to proceed?  
> > What is the next step towards me helping
> > triage issues?  Is there a bridge-keeper with at least three, no more than 
> > five questions for me?
> 
> I would suggest having at least one, if not two or three, current core-devs 
> ready and willing to quickly review your
> work (I believe Raymond Hettinger may be one); then, go ahead and triage, 
> improve and/or submit patches, and make
> comments.  Once you've got a few of these under your belt, with favorable 
> reviews and your patches committed, you may
> get stuck with commit privileges of your own.  ;)

Indeed, the best way to proceed, regardless of any other issues, is in
fact to review, triage, comment on, and improve the issues you are
interested in.  Get the patches commit ready, and then get a current
core dev to do a commit review.

Oddly, the doc issues may be more problematic than the code issues.
Fixing bugs in docs isn't difficult to get done, but restructuring
documentation sometimes gets bogged down in differing opinions.  (I
haven't myself looked at your proposals since I don't use
multiprocessing, so I don't know how radical the proposed changes are).

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[R. David Murray]

On Mon, 12 Jan 2015 17:26:43 +, Steve Dower  
wrote:
> David Anthoff wrote:
> > Yes, those are good examples. Right now doing this in the way these guys do 
> > is
> > too much work for our small project... Anything that makes this easier 
> > would be
> > appreciated.
> 
> I don't see how. All they've done is literally copy a Python
> installation into their install directory. Yes, they have their own
> launcher executables (py2exe generated, it looks like) and have
> precompiled the standard library and put it in a ZIP file, but you
> don't even need to go that far. Without knowing anything about your
> project I can't give specific suggestions, but simply dropping a
> Python installation in is not that difficult (and until the issues
> that Nick referred to are fixed, will have the same problems as
> TortoiseHg presumably has).

That's what py2exe *does*.  It does all the python-integration and
launcher-building work for you.  I use it for a small project myself
(proprietary), which I build an installer for using Inno Setup.  Works
very well, supports python3.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] incorrect docstring for sys.int_info.sizeof_digit?

[R. David Murray]

On Wed, 21 Jan 2015 14:53:19 +, Tim Golden  wrote:
> On 21/01/2015 11:07, Pfeiffer, Phillip E., IV wrote:
> > Apologies if this has already been reported; I couldn't find a
> > readily searchable archive for this mailing list (and apologies if
> > I've just missed it).
> 
> Depending on "readily searchable", this isn't too bad:
> 
> http://markmail.org/search/?q=list%3Aorg.python.python-dev+integer+docstring

But, if you are searching to see if a bug has been reported, you want to
search the tracker at bugs.python.org, not the python-dev mailing list.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] rst files

[R. David Murray]

On Fri, 23 Jan 2015 15:55:29 -0800, Guido van Rossum  wrote:
> This adds entries to the index of the document -- similar to the index at
> the end of a book. I think single vs. double refers to different types of
> entries. Check out this page: https://docs.python.org/3/genindex.html
> 
> On Fri, Jan 23, 2015 at 3:43 PM, Ethan Furman  wrote:
> 
> > Can somebody please explain this?
> >
> > .. index::
> >single: formatting, string (%)
> >single: interpolation, string (%)
> >single: string; formatting
> >single: string; interpolation
> >single: printf-style formatting
> >single: sprintf-style formatting
> >single: % formatting
> >single: % interpolation
> >
> > Specifically, what does index mean?  What does single vs double vs triple
> > mean?  Is there a reference somewhere I can read?

It is explained in the Sphinx documentation:
http://sphinx-doc.org/contents.html

Specifically:

http://sphinx-doc.org/markup/misc.html#directive-index

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why are generated files in the repository?

[R. David Murray]

On Sun, 25 Jan 2015 14:00:57 +1000, Nick Coghlan  wrote:
> It's far more developer friendly to aim to have builds from a source
> check-out "just work" if we can. That's pretty much where we are today
> (getting external dependencies for the optional parts on *nix can still be
> a bit fiddly - it may be worth maintaining instructions for at least apt
> and yum in the developer guide that cover that)

https://docs.python.org/devguide/setup.html#build-dependencies
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any grammar experts?

[R. David Murray]

On Mon, 26 Jan 2015 01:21:24 +0100, Antoine Pitrou  wrote:
> On Sun, 25 Jan 2015 14:59:42 -0800
> Guido van Rossum  wrote:
> > On Sun, Jan 25, 2015 at 7:32 AM, Georg Brandl  wrote:
> > 
> > > On 01/25/2015 04:08 PM, Antoine Pitrou wrote:
> > > > On Sat, 24 Jan 2015 21:10:51 -0500
> > > > Neil Girdhar  wrote:
> > > >> To finish PEP 448, I need to update the grammar for syntax such as
> > > >>
> > > >> {**x for x in it}
> > > >
> > > > Is this seriously allowed by the PEP? What does it mean exactly?
> > >
> > > It appears to go a bit far.  Especially since you also would have to allow
> > >
> > > {*x for x in it}
> > >
> > > which is a set comprehension, while the other is a dict comprehension :)
> > >
> > 
> > That distinction doesn't bother me -- you might as well claim it's
> > confusing that f(*x) passes positional args from x while f(**x) passes
> > keyword args.
> > 
> > And the varargs set comprehension is similar to the varargs list
> > comprehension:
> > 
> > [*x for x in it]
> > 
> > If `it` were a list of three items, this would be the same as
> > 
> > [*it[0], *it[1], *it[2]]
> 
> I find all this unreadable and difficult to understand.

I did too, before reading the PEP.

After reading the PEP, it makes perfect sense to me.  Nor is the PEP
complicated...it's just a matter of wrapping your head around the
generalization[*] of what are currently special cases that is going on
here.

--David

[*] The *further* generalization...we've already had, for example,
the generalization that allows:

a, *b = (1, 3, 4)

to work, and that seems very clear to usnow.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any grammar experts?

[R. David Murray]

On Mon, 26 Jan 2015 09:43:26 -0500, Barry Warsaw  wrote:
> On Jan 25, 2015, at 09:31 PM, R. David Murray wrote:
> 
> >> > > {*x for x in it}
> >> > >
> >> > > which is a set comprehension, while the other is a dict comprehension 
> >> > > :)
> >> > >
> >> > 
> >> > That distinction doesn't bother me -- you might as well claim it's
> >> > confusing that f(*x) passes positional args from x while f(**x) passes
> >> > keyword args.
> >> > 
> >> > And the varargs set comprehension is similar to the varargs list
> >> > comprehension:
> >> > 
> >> > [*x for x in it]
> >> > 
> >> > If `it` were a list of three items, this would be the same as
> >> > 
> >> > [*it[0], *it[1], *it[2]]
> >> 
> >> I find all this unreadable and difficult to understand.
> >
> >I did too, before reading the PEP.
> >
> >After reading the PEP, it makes perfect sense to me.  Nor is the PEP
> >complicated...it's just a matter of wrapping your head around the
> >generalization[*] of what are currently special cases that is going on
> >here.
> 
> It does make sense after reading the PEP but it also reduces the readability
> and instant understanding of any such code.  This is head-scratcher code that
> I'm sure I'd get asked about from folks who aren't steeped in all the dark
> corners of Python.  I don't know if that's an argument not to adopt the PEP,
> but it I think it would be a good reason to recommend against using such
> obscure syntax, unless there's a good reason (and good comments!).

But it is only obscure because we are not used to it yet.  It is a
logical extension of Python's existing conventions once you think about
it.  It will become "obvious" quickly, IMO.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any grammar experts?

[R. David Murray]

On Mon, 26 Jan 2015 22:05:44 +0100, Antoine Pitrou  wrote:
> On Mon, 26 Jan 2015 12:22:20 -0800
> Ethan Furman  wrote:
> > On 01/26/2015 12:09 PM, Antoine Pitrou wrote:
> > > On Mon, 26 Jan 2015 12:06:26 -0800
> > > Ethan Furman  wrote:
> > >> It destroy's the chaining value and pretty much makes the improvement 
> > >> not an improvement.  If there's a possibility that
> > >> the same key could be in more than one of the dictionaries then you 
> > >> still have to do the
> > >>
> > >>   dict.update(another_dict)
> > > 
> > > So what? Is the situation where chaining is desirable common enough?
> > 
> > Common enough to not break it, yes.
> 
> Really? What are the use cases?

My use case is a configuration method that takes keyword parameters.
In tests I want to specify a bunch of default values for the
configuration, but I want individual test methods to be able
to override those values.  So I have a bunch of code that does
the equivalent of:

from test.support import default_config
[...]
def _prep(self, config_overrides):
config = default.config.copy()
config.update(config_overrides)
my_config_object.load(**config)


With the current proposal I could instead do:

def _prep(self, config_overrides):
my_config_object.load(**default_config, **config_overrides)

I suspect I have run into situations like this elsewhere as well, but
this is the one from one of my current projects.

That said, I must admit to being a bit ambivalent about this, since
we otherwise are careful to disallow duplicate argument names.

So, instead we could write:

my_config_object.load(**{**default_config, **config_overrides})

since dict literals *do* allow duplicate keys.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any grammar experts?

[R. David Murray]

On Tue, 27 Jan 2015 00:07:08 +0100, Antoine Pitrou  wrote:
> On Mon, 26 Jan 2015 16:28:24 -0500
> "R. David Murray"  wrote:
> > 
> > My use case is a configuration method that takes keyword parameters.
> > In tests I want to specify a bunch of default values for the
> > configuration, but I want individual test methods to be able
> > to override those values.  So I have a bunch of code that does
> > the equivalent of:
> > 
> > from test.support import default_config
> > [...]
> > def _prep(self, config_overrides):
> > config = default.config.copy()
> > config.update(config_overrides)
> > my_config_object.load(**config)
> > 
> > 
> > With the current proposal I could instead do:
> > 
> > def _prep(self, config_overrides):
> > my_config_object.load(**default_config, **config_overrides)
> 
> It sounds like the _prep() method exists once in your code base, this
> isn't an idiom you are duplicating everywhere. The incentive for a
> syntactic shortcut looks pretty thin.

Something similar exists between five and ten times (I didn't go in and
count) in my current code base, in various specific forms for different
test classes.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TypeError messages

[R. David Murray]

On Sun, 22 Feb 2015 00:26:23 +1000, Nick Coghlan  wrote:
> On 21 February 2015 at 00:05, Brett Cannon  wrote:
> >  I agree that type names don't need to be quoted.
> 
> As a user, I actually appreciate the quotes, because they make the
> error pattern easier for me to parse. Compare:
> 
> int expected, but found str
> float expected, but found int
> 
> 'int' expected, but found 'str'
> 'float' expected, but found 'int'

It's not a big deal to me either way, but I find the version with the
quotes makes me think it is looking for the literal string 'int', but
found the literal string 'str', whereas in the first case it seems
clear it is looking for objects of that type.  Perhaps this is just
because I am used to the existing messages, but I think it goes
beyond that.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Not being able to compile: "make: *** [Programs/_freeze_importlib] Error 1"

[R. David Murray]

On Thu, 16 Apr 2015 18:09:01 -0300, Facundo Batista  
wrote:
> Full trace here:
> 
>   http://linkode.org/TgkzZw90JUaoodvYzU7zX6
> 
> Before going into a deep debug, I thought about sending a mail to see
> if anybode else hit this issue, if it's a common problem, if there's a
> known workaround.

Most likely you just need to run 'make touch' so that it doesn't try
to rebuild stuff it doesn't need to (because we check in those
particular build artifacts, like the frozen importlib).

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of Python tracker Issues

[R. David Murray]

On Fri, 17 Apr 2015 18:08:24 +0200, Python tracker  
wrote:
> 
> ACTIVITY SUMMARY (2015-04-10 - 2015-04-17)
> Python tracker at http://bugs.python.org/
> 
> To view or respond to any of the issues listed below, click on the issue.
> Do NOT respond to this message.
> 
> Issues counts and deltas:
>   open4792 (-31)
>   closed 30957 (+113)
>   total  35749 (+82)

That's a successful sprint week :)

Thanks everyone!

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

I wrote a longer response and then realized it didn't really add much to
the discussion.  So let me be short: type annotations do *not* appeal to
me, and I am not looking forward to the cognitive overhead of dealing
with them.  Perhaps I will eventually grow to like them if the tools
that use them really add value.  You'll have to sell me on it, though.

On Mon, 20 Apr 2015 12:35:33 -0700,  wrote:
> Stub files have many downsides, too, unfortunately:
> - we don’t *want* to have them, but we *need* to have them (C extensions, 
> third-party modules, Python 2, …)
> - they bring cognitive overhead of having to switch between two files
> - they require the author to repeat himself quite a lot
> - they might go out of date much easier than annotations in the function 
> signature

The whole point of type hints is for the linters/IDEs, so IMO it is
perfectly reasonable to put the burden of making them useful onto the
linters/IDEs.  The UI for it can unify the two files into a single
view...I know because way back in the dark ages I wrote a small
editor-based IDE that did something very analogous on an IBM Mainframe,
and it worked really well as a development environment.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

+1 to this from me too. I'm afraid that means I'm -1 on the PEP.

I didn't write this in my earlier email because I wasn't sure about it,
but my gut reaction after reading Harry's email was "if type annotations
are used in the stdlib, I'll probably stop contributing".  That doesn't
mean that's *true*, but that's the first time I've ever had that
thought, so it is probably worth sharing.

Basically, it makes Python less fun to program in.  That may be be an
emotional reaction and irrational, but I think it matters.  And yes, I
write production Python code for a living, though granted not at Google
or Facebook or Dropbox scale.

On Mon, 20 Apr 2015 19:00:53 -0500, Ryan Gonzalez  wrote:
> Although I like the concept of type annotations and the PEP, I have to
> agree with this. If I saw these type annotations when learning Python (I'm
> self-taught), there's a 99% chance I would've freaked.
> 
> It's the same issue as with teaching C++: it's wrong to say, "Hey, I taught
> you the basics, but there's other stuff that's going to confuse you to a
> ridiculous extent when you read it." People can't ignore it. It'll become a
> normal part of Python programs.
> 
> At least now you can say, "I'm using the mypy type checker."
> 
> Don't get me wrong; I like mypy. I helped with their documentation and am
> watching the GitHub repo. But this is dead-on.
> 
> 
> On Mon, Apr 20, 2015 at 6:41 PM, Jack Diederich  wrote:
> 
> > Twelve years ago a wise man said to me "I suggest that you also propose a
> > new name for the resulting language"
> >
> > I talked with many of you at PyCon about the costs of PEP 484. There are
> > plenty of people who have done a fine job promoting the benefits.
> >
> > * It is not optional. Please stop saying that. The people promoting it
> > would prefer that everyone use it. If it is approved it will be optional in
> > the way that PEP8 is optional. If I'm reading your annotated code it is
> > certainly /not/ optional that I understand the annotations.
> >
> > * Uploading stubs for other people's code is a terrible idea. Who do I
> > contact when I update the interface to my library? The random Joe who
> > "helped" by uploading annotations three months ago and then quit the
> > internet? I don't even want to think about people maliciously adding stubs
> > to PyPI.
> >
> > * The cognitive load is very high. The average function signature will
> > double in length. This is not a small cost and telling me it is "optional"
> > to pretend that every other word on the line doesn't exist is a farce.
> >
> > * Every company's style guide is about to get much longer. That in itself
> > is an indicator that this is a MAJOR language change and not just some
> > "optional" add-on.
> >
> > * People will screw it up. The same people who can't be trusted to program
> > without type annotations are also going to be *writing* those type
> > annotations.
> >
> > * Teaching python is about to get much less attractive. It will not be
> > optional for teachers to say "just pretend all this stuff over here doesn't
> > exist"
> >
> > * "No new syntax" is a lie. Or rather a red herring. There are lots of new
> > things it will be required to know and just because the compiler doesn't
> > have to change doesn't mean the language isn't undergoing a major change.
> >
> > If this wasn't in a PEP and it wasn't going to ship in the stdlib very few
> > people would use it. If you told everyone they had to install a different
> > python implementation they wouldn't. This is much worse than that - it is
> > Python4 hidden away inside a PEP.
> >
> > There are many fine languages that have sophisticated type systems. And
> > many bondage & discipline languages that make you type things three times
> > to make really really sure you meant to type that. If you find those other
> > languages appealing I invite you to go use them instead.
> >
> > -Jack
> >
> > https://mail.python.org/pipermail/python-dev/2003-February/033291.html
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
> >
> >
> 
> 
> -- 
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something’s wrong.
> http://kirbyfan64.github.io/
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

On Wed, 22 Apr 2015 01:09:52 +1000, Chris Angelico  wrote:
> def incremental_parser(input: FileLike) -> List[Token]:
> tokens = []
> data = ""
> while True:
> if not data:
> data = input.read(64)
> token = Token(data[0]); data = data[1:]
> while token.wants_more():
> token.give_more(data[0]); data = data[1:]
> tokens.append(token)
> if token.is_end_of_stream(): break
> input.seek(-len(data), 1)
> return tokens
> 
> If you were to exhaustively stipulate the requirements on the
> file-like object, you'd have to say:
> 
> * Provides a read() method which takes an integer and returns up to
> that many bytes
> * Provides a seek() method which takes two integers
> * Is capable of relative seeking by at least 63 bytes backward
> * Is open for reading
> * Etcetera
> 
> That's not the type system's job. Not in Python. Maybe in Haskell, but

Just a note that if I'm reading the high level description right,
this kind of analysis is exactly the kind of thing that Flow does
for javascript.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

On Tue, 21 Apr 2015 18:27:50 +0300, Paul Sokolovsky  wrote:
> > I was replying to Steven's message. Did you read it?
> 
> Yes. And I try to follow general course of discussion, as its hard to
> follow individual sub-threads. And for example yesterday's big theme
> was people blackmailing that they stop contributing to stdlib if
> annotations are in, and today's theme appear to be people telling that
> static type checking won't be useful. And just your reply to Steven
> was a final straw which prompted me to point out that static type
> checking is not a crux of it, but just one (not the biggest IMHO) usage.

Please be respectful rather than inflammatory.  If you read what I
wrote, I did not say that I was going to stop contributing, I
specifically talked about that gut reaction being both emotional and
illogical.  That doesn't make the reaction any less real, and the fact
that such reactions exist is a data point you should consider in
conducting your PR campaign for this issue.  (I don't mean that last as
a negative:  this issue *requires* an honest PR campaign.)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

On Tue, 21 Apr 2015 16:55:49 -, "Gregory P. Smith"  wrote:
> We will not be putting type annotations anywhere in the stdlib or expecting
> anyone else to maintain them there. That would never happen until tools
> that are convincing enough in their utility for developers to _want_ to use
> are available and accepted.  That'll be a developer workflow thing we could
> address with a later PEP. IF it happens at all.

This is the most reassuring statement I've heard so far ;)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

On Tue, 21 Apr 2015 10:10:06 -0700, Guido van Rossum  wrote:
> On Tue, Apr 21, 2015 at 9:17 AM, R. David Murray 
> wrote:
> 
> > Please be respectful rather than inflammatory.  If you read what I
> > wrote, I did not say that I was going to stop contributing, I
> > specifically talked about that gut reaction being both emotional and
> > illogical.  That doesn't make the reaction any less real, and the fact
> > that such reactions exist is a data point you should consider in
> > conducting your PR campaign for this issue.  (I don't mean that last as
> > a negative:  this issue *requires* an honest PR campaign.)
> >
> 
> Well, my own reactions at this point in the flame war are also quite
> emotional. :-(
> 
> I have done my best in being honest in my PR campaign. But I feel like the

I believe you have.  My inclusion of the word 'honest' was meant to
contrast the kind of PR we need with the kind of PR people typically
think about, which is often not particularly honest.

> opposition (not you, but definitely some others -- have you seen Twitter?)
> are spreading FUD based on an irrational conviction that this will destroy

No, I tend only to peek in there occasionally.

> Python. It will not. It may not prove the solution to all Python's problems
> -- there's always 3.6. (Oh wait, Python 2.7 is perfect. I've heard that
> before -- Paul Everitt famously said the same of Python 1.5.2. Aren't you
> glad I didn't take him literally? :-P )

Yes.  But somewhere there or not long before was my introduction to
Python, so I remember it fondly :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

On Tue, 21 Apr 2015 21:31:49 +0300, Paul Sokolovsky  wrote:
> On Tue, 21 Apr 2015 09:50:59 -0700 Ethan Furman  wrote:
> 
> > On 04/21, Paul Sokolovsky wrote:
> > > 
> > > And for example yesterday's big theme was people blackmailing that
> > > they stop contributing to stdlib if annotations are in [...]
> > 
> > A volunteer's honest reaction is not blackmail, and categorizing it
> > as such is not helpful to the discussion.
> 
> Sure, that was rather humoresque note. Still, one may wonder why
> "honest reaction" is like that, if from reading PEP484 it's clear that
> it doesn't change status quo: https://www.python.org/dev/peps/pep-3107
> added annotations long ago, and PEP484 just provides default

But what concerned me as a Python core developer was the perception that
as a core developer I would have to deal with type hints and type
checking in the stdlib, because there was talk of including type hints
for the stdlib (as stub files) in 3.6.  *That* was the source of my
concern, which is reduced by the statement that we'll only need to think
about type checking in the stdlib once the tools are *clearly* adding
value, in *our* (the stdlib module maintainers') opinion, so that
we *want* to use the type hints.

The discussion of stub files being maintained by interested parties in a
separate repository is also helpful.  Again, that means that wider
adoption should mostly only happen if the developers see real benefit.
I still dislike the idea of having to read type hints (I agree with
whoever it was said a well written docstring is more helpful than
abstract type hints when reading Python code), but perhaps I will get
used to it if/when it shows up in libraries I use.

I think it would serve your goal better if instead of dismissing
concerns you were to strive to understand them and then figure out how
to allay them.

Overall, I suspect that those who are doubtful are put off by the "rah
rah this is great you are fools for not wanting to use it" tone of
(some) of the proponents (which makes us think this is going to get
pushed on us willy-nilly), whereas hearing a consistent message that
"this will *enable* interested parties to collaborate on type checking
without impacting the way the rest of you code *unless it proves its
value to you*" would be better received.  Guido has done the latter,
IMO, as have some others.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allow annotations using basic types in the stdlib?

[R. David Murray]

I agree with Steve.  There is *cognitive* overhead to type annotations.
I find that they make Python code harder to read and understand.  So I
object to them in the documentation and docstrings as well.  (Note:
while I agree that the notation is compact for the simple types, the
fact that it would appear for some signatures and not for others is a
show stopper from my point of view...consistency is important to reducing
the cognitive overhead of reading the docs.)

I'm dealing with the spread of annotations on my current project,
having to ask programmers on the team to delete annotations that they've
"helpfully" added that to my mind serve no purpose on a project of the
size we're developing, where we aren't using static analysis for anything.

Maybe I'm being a curmudgeon standing in the way of progress, but I'm
pretty sure there are a number of people in my camp :)

On Mon, 06 Nov 2017 16:22:23 +, Steve Holden  wrote:
> While I appreciate the value of annotations I think that *any* addition of
> them to the stdlib would complicate an important learning resource
> unnecessarily. S
> 
> Steve Holden
> 
> On Mon, Nov 6, 2017 at 4:07 PM, Victor Stinner 
> wrote:
> 
> > Related to annotations, are you ok to annotate basic types in the
> > *documentation* and/or *docstrings* of the standard library?
> >
> > For example, I chose to document the return type of time.time() (int)
> > and time.time_ns() (float). It's short and I like how it's formatted.
> > See the current rendered documentation:
> >
> > https://docs.python.org/dev/library/time.html#time.time
> >
> > "Annotations" in the documentation and docstrings have no impact on
> > Python runtime performance. Annotations in docstrings makes them a few
> > characters longer and so impact the memory footprint, but I consider
> > that the overhead is negligible, especially when using python3 -OO.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (3.4): Issue #23840: tokenize.open() now closes the temporary binary file on error to

[R. David Murray]

On Tue, 26 May 2015 08:20:01 +0200, Victor Stinner  
wrote:
> What is wrong with "except:" in this specific case?

Nothing is wrong with it from a technical standpoint.  However, if we
use 'except BaseException' that makes it clear that someone has thought
about it and decided that all exceptions should be caught, as opposed to
it being legacy code or a programming mistake.

> > On 2015-05-26 12:26 AM, Terry Reedy wrote:
> >
> >>> +try:
> >>> +encoding, lines = detect_encoding(buffer.readline)
> >>> +buffer.seek(0)
> >>> +text = TextIOWrapper(buffer, encoding, line_buffering=True)
> >>> +text.mode = 'r'
> >>> +return text
> >>> +except:
> >>> +buffer.close()
> >>> +raise
> >>>
> >>
> >> Please do not add bare 'except:'.  If you mean 'except BaseException:',
> >> say so.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 is here until 2020, please don't call it a waste.

[R. David Murray]

On Wed, 03 Jun 2015 12:04:10 +0200, Maciej Fijalkowski  wrote:
> On Wed, Jun 3, 2015 at 11:38 AM, M.-A. Lemburg  wrote:
> > On 02.06.2015 21:07, Maciej Fijalkowski wrote:
> >> Hi
> >>
> >> There was a PSF-sponsored effort to improve the situation with the
> >> https://bitbucket.org/pypy/codespeed2/src being written (thank you
> >> PSF). It's not better enough than codespeed that I would like, but
> >> gives some opportunities.
> >>
> >> That said, we have a benchmark machine for benchmarking cpython and I
> >> never deployed nightly benchmarks of cpython for a variety of reasons.
> >>
> >> * would be cool to get a small VM to set up the web front
> >>
> >> * people told me that py3k is only interesting, so I did not set it up
> >> for py3k because benchmarks are mostly missing
> >>
> >> I'm willing to set up a nightly speed.python.org using nightly build
> >> on python 2 and possible python 3 if there is an interest. I need
> >> support from someone maintaining python buildbot to setup builds and a
> >> VM to set up stuff, otherwise I'm good to go
> >>
> >> DISCLAIMER: I did facilitate in codespeed rewrite that was not as
> >> successful as I would have hoped. I did not receive any money from the
> >> PSF on that though.
> >
> > I think we should look into getting speed.python.org up and
> > running for both Python 2 and 3 branches:
> >
> >  https://speed.python.org/
> >
> > What would it take to make that happen ?
> 
> I guess ideal would be some cooperation from some of the cpython devs,
> so say someone can setup cpython buildbot

What does "set up cpython buildbot" mean in this context?

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] speed.python.org (was: 2.7 is here until 2020, please don't call it a waste.)

[R. David Murray]

On Thu, 04 Jun 2015 12:55:55 +0200, "M.-A. Lemburg"  wrote:
> On 04.06.2015 04:08, Tetsuya Morimoto wrote:
> >> If someone were to volunteer to set up and run speed.python.org, I think
> > we could add some additional focus on performance regressions. Right now,
> > we don't have any way of reliably and reproducibly testing Python
> > performance.
> > 
> > I'm very interested in speed.python.org and feel regret that the project is
> > standing still. I have a mind to contribute something ...
> 
> On 03.06.2015 18:59, Maciej Fijalkowski wrote:> On Wed, Jun 3, 2015 at 3:49 
> PM, R. David Murray
> >>>> I think we should look into getting speed.python.org up and
> >>>> running for both Python 2 and 3 branches:
> >>>>
> >>>>  https://speed.python.org/
> >>>>
> >>>> What would it take to make that happen ?
> >>>
> >>> I guess ideal would be some cooperation from some of the cpython devs,
> >>> so say someone can setup cpython buildbot
> >>
> >> What does "set up cpython buildbot" mean in this context?
> >
> > The way it works is dual - there is a program running the benchmarks
> > (the runner) which is in the pypy case run by the pypy buildbot and
> > the web side that reports stuff. So someone who has access to cpython
> > buildbot would be useful.

(I don't seem to have gotten a copy of Maciej's message, at least not
yet.)

OK, so what you are saying is that speed.python.org will run a buildbot
slave so that when a change is committed to cPython, a speed run will be
triggered?  Is "the runner" a normal buildbot slave, or something
custom?  In the normal case the master controls what the slave
runs...but regardless, you'll need to let us know how the slave
invocation needs to be configured on the master.

> Ok, so there's interest and we have at least a few people who are
> willing to help.
> 
> Now we need someone to take the lead on this and form a small
> project group to get everything implemented. Who would be up
> to such a task ?
> 
> The speed project already has a mailing list, so you could use
> that for organizing the details.

If it's a low volume list I'm willing to sign up, but regardless I'm
willing to help with the buildbot setup on the CPython side.  (As soon
as my credential-update request gets through infrastructure, at least :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker reviews look like spam

[R. David Murray]

On Fri, 22 May 2015 08:05:49 +0300, Antti Haapala  wrote:
> There's an issue about this at
> http://psf.upfronthosting.co.za/roundup/meta/issue562
> 
> I believe the problem is not that of the SPF, but the fact that mail gets
> sent using IPv6 from an address that has neither a name mapping to it nor a
> reverse pointer from IP address to name in DNS. See the second-first
> comment where R. David Murray states that "Mail is consistently sent from

The ipv6 reverse dns issue is being worked on.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker reviews look like spam

[R. David Murray]

On Tue, 09 Jun 2015 21:41:23 -, "Gregory P. Smith"  wrote:
> I *believe* you can get this to happen today in a review if you add the
> rep...@bugs.python.org email address to the code review as the issueX
> in the subject line will make the tracker turn it into a bug comment.  If
> so, having that be the default cc for all reviews created would be a great
> feature (and modify it not to send mail to anyone else).

I haven't double checked, but I think the issue number has to be in
square brackets to be recognized.  Presumably that's a change that
could be made.  What is lacking is someone willing to climb the
relatively steep learning curve needed to submit patches for that
part of the system, and some one of us with the keys to the tracker
with time to apply the patch.  Given the former I think we can
manage the latter.

I believe Ezio did try to make rietveld update the tracker, and ran
into a problem whose nature I don't know...but I don't think he spent
a whole lot of time trying to debug the problem, whatever it was.
I imagine he'll chime in :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Importance of "async" keyword

[R. David Murray]

On Sat, 27 Jun 2015 01:10:33 +1000, Chris Angelico  wrote:
> The way I'm seeing it, coroutines are like cooperatively-switched
> threads; you don't have to worry about certain operations being
> interrupted (particularly low-level ones like refcount changes or list
> growth), but any time you hit an 'await', you have to assume a context
> switch. That's all very well, but I'm not sure it's that big a problem
> to accept that any call could context-switch; atomicity is already a
> concern in other cases, which is why we have principles like EAFP
> rather than LBYL.

Read Glyph's article, it explains why:

https://glyph.twistedmatrix.com/2014/02/unyielding.html

> There's clearly some benefit to being able to assume that certain
> operations are uninterruptible (because there's no 'await' anywhere in
> them), but are they truly so? Signals can already interrupt something
> _anywhere_:

Yes, and you could have an out of memory error anywhere in your program
as well.  (Don't do things in your signal handlers, set a flag.)  But
that doesn't change the stuff Glyph talks about (and Guido talks about)
about *reasoning* about your code.

I did my best to avoid using threads, and never invested the time and
effort in Twisted.  But I love programming with asyncio for highly
concurrent applications.  It fits in my brain :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should asyncio ignore KeyboardInterrupt?

[R. David Murray]

Once long ago in Internet time (issue 581232) time.sleep on windows was
not interruptible and this was fixed.  Is it possible the work on EINTR
has broken that fix?

(I don't currently have 3.5 installed on windows to test that theory...)

On Sat, 04 Jul 2015 17:46:34 +0200, Guido van Rossum  wrote:
> I think this may be more of a Windows issue than an asyncio issue. I agree
> that ideally ^C should take effect immediately (as it does on UNIX).
> 
> On Sat, Jul 4, 2015 at 9:54 AM, Terry Reedy  wrote:
> 
> > Should the loop.run... methods of asyncio respect KeyboardInterrupt (^C)?
> >
> > Developer and user convenience and this paragraph in PEP
> >
> > "However, exceptions deriving only from BaseException are typically not
> > caught, and will usually cause the program to terminate with a traceback.
> > In some cases they are caught and re-raised. (Examples of this category
> > include KeyboardInterrupt and SystemExit ; it is usually unwise to treat
> > these the same as most other exceptions.) "
> >
> > and this examples in the doc (two places)
> >
> > TCP echo server
> > # Serve requests until CTRL+c is pressed
> > print('Serving on {}'.format(server.sockets[0].getsockname()))
> > try:
> > loop.run_forever()
> > except KeyboardInterrupt:
> > pass
> >
> > suggest yes.  On the other hand, the section on
> > "Set signal handlers for SIGINT and SIGTERM"
> > suggests not, unless an explicit handler is registered and then only on
> > Unix.
> >
> > In any case, Adam Bartos, python-list, "An asyncio example", today asks.
> > '''
> > This is a minimal example:
> >
> > import asyncio
> >
> > async def wait():
> > await asyncio.sleep(5)
> >
> > loop = asyncio.get_event_loop()
> > loop.run_until_complete(wait())
> >
> > Ctrl-C doesn't interrupt the waiting, instead KeyboardInterrupt occurs
> > after those five seconds. It's 3.5.0b2 on Windows. Is it a bug?
> > '''
> >
> > Using run_forever instead, I found no way to stop other than killing the
> > process (Idle or Command Prompt).
> >
> > --
> > Terry Jan Reedy
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] What's New editing

[R. David Murray]

Just so people aren't caught unawares, it is very unlikely that I will have
time to be the final editor on "What's New for 3.5" they way I was for 3.3 and
3.4.  I've tried to encourage people to keep What's New up to date, but
*someone* should make a final editing pass.  Ideally they'd do at least the
research Serhiy did last year on checking that there's a mention for all of the
versionadded and versionchanged 3.5's in the docs.  Even better would be to
review the NEWS and/or commit history...but *that* is a really big job these
days

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New editing

[R. David Murray]

On Mon, 06 Jul 2015 11:06:41 +1000, Nick Coghlan  wrote:
> On 6 July 2015 at 03:52, R. David Murray  wrote:
> > Just so people aren't caught unawares, it is very unlikely that I will have
> > time to be the final editor on "What's New for 3.5" they way I was for 3.3 
> > and
> > 3.4.
> 
> And thank you again for your work on those!
> 
> > I've tried to encourage people to keep What's New up to date, but
> > *someone* should make a final editing pass.  Ideally they'd do at least the
> > research Serhiy did last year on checking that there's a mention for all of 
> > the
> > versionadded and versionchanged 3.5's in the docs.  Even better would be to
> > review the NEWS and/or commit history...but *that* is a really big job these
> > days
> 
> What would your rough estimate of the scope of work be? As you note,
> the amount of effort involved in doing a thorough job of that has
> expanded beyond what can reasonably be expected of volunteer
> contributors, so I'm wondering if it might make sense for the PSF to
> start offering a contract technical writing gig to finalise the What's
> New documentation for each new release.
> 
> After all, the What's New doc is an essential component of
> communicating changes in recommended development practices to Python
> educators, so ensuring we do a good job with that can have a big
> multiplier effect on all the other work that goes into creating each
> new release.

I can tell you that 3.4 took me approximately 67 hours according to my
time log.  That was going through the list prepared by Serhiy, and going
through pretty much all of the NEWS entries but not the commit log.  I'm
a precisionist, so I suspect someone less...ocd...about the details
could do it a bit faster, perhaps at the cost of some small amount of
accuracy :)

On the other hand, my knowledge of the code base and the development
that had been going on probably sped up my analysis and writeup of
the missing entries (and revision of existing entries, in many cases).

On gripping hand, I also did some small amount of documentation
rewriting and clarification along the way.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New editing

[R. David Murray]

On Mon, 06 Jul 2015 21:45:01 +0300, Serhiy Storchaka  
wrote:
> On 05.07.15 20:52, R. David Murray wrote:
> > Just so people aren't caught unawares, it is very unlikely that I will have
> > time to be the final editor on "What's New for 3.5" they way I was for 3.3 
> > and
> > 3.4.  I've tried to encourage people to keep What's New up to date, but
> > *someone* should make a final editing pass.  Ideally they'd do at least the
> > research Serhiy did last year on checking that there's a mention for all of 
> > the
> > versionadded and versionchanged 3.5's in the docs.  Even better would be to
> > review the NEWS and/or commit history...but *that* is a really big job these
> > days
> 
> Many thanks you David for your invaluable work.
> 
> Here is 3.5 NEWS file cleaned from duplicates in 3.4 NEWS file (i.e. 
> from entries about merged bug fixes). It is much less than unfiltered 
> NEWS file. Hope this will help volunteers.

That's great.  What I did was work from the html-rendered NEWS page, and
click through to the issue to figure out whether it was a bug fix or an
enhancement.  Not having to do that check should speed things up.  I
seem to recall I did find a couple of things that were screwed up and
still bore mentioning in whatsnew, but I doubt that is likely enough to
make enough difference to be worth it.  I also wound up fixing some
incorrect NEWS entries (wrong numbers, English, other errors), but that
is not central to the whatsnew project.  That activity was probably
included in the hours count, though.

For David (or whoever):  in addition to the obvious task of writing up
appropriate entries in What's New, part of what I did was to make sure
that all of the relevant documentation entries had the appropriate
versionchanged or versionadded tags, and that the new documentation made
sense.  As I recall, my working rhythm was to write the What's New entry
including links to the things that had changed, render the what's new
page to html, fix the links, then work through the links to make sure
the docs made sense and there were appropriate 'versionxxx' tags.  You,
of course, may find a different working style more beneficial :).

Oh, and work from newest change to oldest change.  I did it from oldest
to newest and only realized late in the game that was the wrong order,
because some changes got undone or modified by later changes :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Freeze exception for http://bugs.python.org/issue23661 ?

[R. David Murray]

On Tue, 14 Jul 2015 14:01:25 +1200, Robert Collins  
wrote:
> So unittest.mock regressed during 3.5, and I found out when I released
> the mock backport.
> 
> The regression is pretty shallow - I've applied the fix to 3.6, its a
> one-liner and comes with a patch.
> 
> Whats the process for getting this into 3.5? Its likely to affect a
> lot of folk using mock (pretty much every OpenStack project got git
> with it when I released mock 1.1).

3.5 hasn't been released yet.  The patch ideally would have gone into
3.5 first, then been merged to 3.6.  As it is, you'll apply it to
3.5, and then do a null merge to 3.6.  It will get released in the
next 3.5 beta.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Cross compiling C-python 2.7.10 maintenance release for ARM on 64 bit x86_64 systems.

[R. David Murray]

On Tue, 14 Jul 2015 10:22:05 -, Andrew Robinson  
wrote:
> I'm trying to cross compile C-python 2.7.10 for an embedded system. (Eg: 
> a Kobo reader).
> But there appears to be some bugs that do not allow the latest 
> maintenance release of Python to correctly cross compile on an x86-64 
> build system, for a 32 bit arm system.

To my understanding we don't yet fully support this (though we'd like
to), because we don't have a buildbot that regularly does cross compiles.
There are open issues in the tracker, perhaps you can vet and/or submit
some patches.[*]  Or contribute a buildbot?

--David

[8} See eg http://bugs.python.org/issue5404; I'm guessing there are others.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Where are bugs with the web site reported?

[R. David Murray]

On Thu, 16 Jul 2015 12:24:45 -0700, Glenn Linderman  
wrote:
> On 7/16/2015 12:11 PM, Ryan Gonzalez wrote:
> > I have encountered this weird issue on Chrome for Android where 
> > scrolling up just a little causes the page to dart to the top. I was 
> > going to report it in the bug tracker, but I didn't see a label for 
> > the web site itself.
> >
> > Worst part is, this is stopping me from reading the humor page!
> 
> Sounds more like a bug in Chrome than on the site, unless it is 
> repeatable using other browsers, or unless the site has Chrome-specific 
> code.

python.org bugs are *not* reported on bugs.python.org.  I don't remember
where they are reported...it's on github somewhere I think.

The fact that it isn't obvious may be a good candidate for a bug
report :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Burning down the backlog.

[R. David Murray]

On Sun, 26 Jul 2015 22:59:51 +0100, Paul Moore  wrote:
> On 26 July 2015 at 16:39, Berker Peksağ  wrote:
> >> I'm not actually clear what "Commit Review" status means. I did do a
> >> quick check of the dev guide, and couldn't come up with anything,
> >
> > https://docs.python.org/devguide/triaging.html#stage
> 
> Thanks, I missed that. The patches I checked seemed to have been
> committed and were still at commit review, though. Doesn't the roundup
> robot update the stage when there's a commit? (Presumably not, and
> people forget to do so too).

Yes, it is manual.  Making it automatic would be nice.  Patches accepted
:) Writing a Roundup detector for this shouldn't be all that hard once
you figure out how detectors work.  See:


http://www.roundup-tracker.org/docs/customizing.html#detectors-adding-behaviour-to-your-tracker

The steep part of the curve there is testing your work, which is
something some effort has been made to simplify, but unfortunately I'm
not up on the details of that currently.

In the meantime, this is a service triagers could perform: look at the
commit review issues and make sure that really is the correct stage.

Now, as for the original question:

First, a little history so that the old timers and the newer committers
are on the same page.  When 'commit review' was originally introduced,
what it was used for was for what was then a "second committer" required
review during the RC phase.  I have recently (last two years?) with
agreement of the workflow list and with no objection from committers
shifted this to the model documented in the devguide currently.

However, I agree that what is currently in the devguide is not
sufficient.  Here is my actual intent for the workflow:

1) Issue is opened.  Triager/committer sets it to 'patch needed' if they
believe the bug should be fixed/feature implemented.  (A committer may
choose to override a triager decision and close the bug, explaining why
for the benefit of the triager and all onlookers.)

2) Once we have a patch, one or more triage or committer people work
with the submitter or whoever is working on the patch (who may have no
special role or be a triager or be a committer) in a patch
review-and-update cycle until a triager or a committer thinks it is
ready for commit.

3) If the patch was submitted and/or worked on by a committer, the patch
can be committed.

4) If the patch is not by a committer, the stage should be set to
'commit review' by a triager.  Committers with time available should, as
Robert suggests, look for patches in 'commit review' status *and review
them*.  The wording of "a quick once over" in the devguide isn't
*wrong*, but it does give the impression the patch is "ready to commit",
whereas the goal here is to review the work of the *triager*.  If the
patch is not actually commit ready for whatever reason, it gets bounced
back to patch review with an explanation as to why.

5) Eventually (hopefully the first time or quickly thereafter most of
the time!) the patch really is ready to commit and gets committed.

An here, to my mind, is the most important bit:

6) When the patches moved to commit ready by a given triager are
consistently committable after the step 4 review, it is time to offer
that triager commit privileges.

My goal here is to *transfer* the knowledge of what makes a good review
process and a good patch from the existing committers to new committers,
and therefore acquire new committers.

Now, the problem that Paul cites about not feeling comfortable with the
*commit* process is real (although I will say that at this point I can
go months without doing a commit and I still remember quite clearly how
to do one; it isn't that complicated).  Improving the tooling is one way
to attack that.  I think there can be two types of tooling:  the large
scale problem the PEPs are working toward, and smaller scale helper
scripts such as Paul mentions that one or more committers could develop
and publish (patchcheck on steroids).

Before that, though, it is clear that the devguide needs a "memory
jogger" cheat sheet on how to do a multi-branch commit, linked from
the quicklinks section.

So, I'm hoping Carol will take what I've written above and turn it into
updates for the devguide (assuming no one disagrees with what I've said :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[R. David Murray]

On Mon, 27 Jul 2015 02:09:19 -0500, Tim Peters  wrote:
> Seriously, try this exercise:  how would you code Paul's example if
> "your kind" of arithmetic were in use instead?  For a start, you have
> no idea in advance how many hours you may need to add to get to "the
> same local time tomorrow".  24 won't always work  Indeed, no _whole_
> number of hours may work (according to one source I found, Australia's
> Lord Howe Island uses a 30-minute DST adjustment).  So maybe you don't
> want to do it by addition.  What then?  Pick apart the year, month and
> day components, then simulate "naive arithmetic" by hand?
> 
> The point is that there's no _obvious_ way to do it then.  I'd
> personally strip off the tzinfo member, leaving a wholly naive
> datetime where arithmetic "works correctly" ;-) , add the day, then
> attach the original tzinfo member again.

I *think* I'd be happy with that solution.

I'm not sure if the opinion of a relatively inexperienced timezone user
(whose head hurts when thinking about these things) is relevant, but in
case it is:

My brief experience with pytz is that it gets this all "wrong".  (Wrong
is in quotes because it isn't wrong, it just doesn't match my use
cases).  If I add a timedelta to a pytz datetime that crosses the DST
boundary, IIRC I get something that still claims it is in the previous
"timezone" (which it therefore seemed to me was really a UTC offset),
and I have to call 'normalize' to get it to be in the correct "timezone"
(UTC offset).  I don't remember what that does to the time, and I have
no intuition about it (I just want it to do the naive arithmetic!)

This makes no sense to me, since I thought a tzinfo object was supposed
to represent the timezone including the DST transitions.  I presume this
comes from the fact that datetime does naive arithmetic and pytz is
trying to paste non-naive arithmetic on top?

So, would it be possible to take the timezone database support from
pytz, and continue to implement naive-single-zone arithmetic the way Tim
proposes, and have it automatically produce the correct UTC offset and
UTC-offset-label afterward, without a normalize call?  I assumed that
was what the PEP was proposing, but I never read it so I can't complain
that I was wrong :)

I have a feeling that I'm completely misunderstanding things, since
tzinfo is still a bit of a mystery to me.

Based on this discussion it seems to me that (1) datetime has to/should
continue to do naive arithmetic and (2) if you need to do non-naive UTC
based calculations (or conversions between timezones) you should be
converting to UTC *anyway* (explicit is better than implicit).  The
addition of the timezone DB would then give us the information to *add*
tools that do conversions between time zones &c.

At Tim says, the issue of disambiguation is separate, and I seem to
recall a couple of proposals from the last time this thread went around
for dealing with that.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com