[Python-Dev] M.L.S.(Multiple Language Support) in Core-Compiler

[R.]

How to adapt the existing M.L.S. from Compilermessages to
Core-Compiler-Words.

Where to begin, for implemantation of this feature?



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] (no subject)

[R Alshammrei]

--

في 8 سبتمبر ، 2017 في تمام الساعة 12:30 بعد منتصف الليل ، سيقوم
Masayuki YAMAMOTO
< ma3yuki.8mamo10 at gmail.com
 > كتب:
> *مرحبا يا قوم،
* > > *I إرسال PEP 539 المسودة الثالثة للالنهاية. شكرا لجميع النصائح
* > *والمساعدة!
*
شكرا لك إريك وياماموتو سان لجميع عملك على هذا PEP!
تبدو التحديثات جيدة ، لذلك يسعدني أن أقول إن BDFL-Delegate ذلك
الاقتراح مقبول الآن :)
مع تحياتي،
نيك.
-
نيك كوغلان   ncoghlan في gmail.com
| بريسبان،
أستراليا

--


   - Previous message (by thread): [Python-Dev] PEP 539 v3: A C new API for
   Thread-Local Storage in CPython
   
   - الرسالة التالية (بواسطة مؤشر الترابط): [Python-Dev] PEP 539 v3: واجهة
   برمجة تطبيقات C جديدة للتخزين المحلي الخيطي في CPython
   
   - *تم فرز الرسائل حسب: *[date]
   

   [thread]
   

   [subject]
   

   [author]
   


--
مزيد من المعلومات حول القائمة البريدية لـ Python-Dev

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 3.0.1 and mingw

[Vincent R.]

Hi,

I wanted to know if you have some patch to compile python 3.x on mingw
platform because I found some
but doesn't work very well :

> make
gcc   -o python.exe \
Modules/python.o \
libpython3.0.a-lm
Could not find platform independent libraries 
Could not find platform dependent libraries 
Consider setting $PYTHONHOME to [:]
Fatal Python error: Py_Initialize: can't initialize sys standard streams
ImportError: No module named encodings.utf_8


I have some questions about posixmodule.c, config.c and makesetup, 
I can see in posixmodule that you define a INITFUNC like this :

#if (defined(_MSC_VER) || defined(__WATCOMC__) || defined(__BORLANDC__)) &&
!defined(__QNX__)
#define INITFUNC PyInit_nt
#define MODNAME "nt"

#elif defined(PYOS_OS2)
#define INITFUNC PyInit_os2
#define MODNAME "os2"

#else
#define INITFUNC PyInit_posix
#define MODNAME "posix"
#endif

So first I tried to add || defined(MINGW32) to declare a PyInit_nt
but after config.c
was still using PyInit_posix. How does makesetup choose to include one
function or another ?
So finally python is using PyInit_posix...


Any idea why I got this compilation error ?


And after 



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should there be a way or API for retrieving from a code object a loader method and package file where the code comes from?

[R. Bernstein]

Paul Moore writes:
 > 2008/12/23  :
 > > What is wanted is a uniform way get and describe a file location
 > > from a code object that takes into account the file might be a member
 > > of an archive.
 > 
 > But a code object may not have come from a file. 

Right. That's why I mentioned for example "eval" and "exec" that you
cite below. So remove the "file" in what is cited above. Replace with:
"a unform way to get information (not necessarily just the source
text) about the location/origin of code from a code object.

 > Ignoring the
 > interactive prompt (not because it's unimportant, just because people
 > have a tendency to assume it's the only special case :-)) you need to
 > consider code loaded via a PEP302 importer from (say) a sqlite
 > database, or code created using compile(), or possibly even more
 > esoteric means.
 > 
 > So I'm not sure your request is clearly specified.

Is the above any more clear? 

 > 
 > > Are there even guidelines for saying what string goes into a code
 > > object's co_filename? Clearly it should be related to the source code
 > > that generated the code, and there are various conventions that seem
 > > to exist when the code comes from an "eval" or an "exec".
 > 
 > I'm not aware of guidelines - the documentation for compile() says
 > "The filename argument should give the file from which the code was
 > read; pass some recognizable value if it wasn't read from a file
 > ('' is commonly used)" which is pretty non-commital.
 > 
 > > But empirically it seems as though there's some variation. It could be
 > > an absolute file or a file with no root directory specified. (But is
 > > it possible to have things like "." and ".."?). And in the case of a
 > > member of a package what happens? Should it be just the member without
 > > the package? Or should it include the package name like
 > >   /usr/lib/python2.5/site-packages/tracer-0.1.0-py2.5.egg/tracer.py ?
 > >
 > > Or be unspecified? If left unspecified as I gather it is now, it makes
 > > it more important to have some sort of common routine to be able to
 > > pick out the archive part in a filesystem from the member name inside
 > > the archive.
 > 
 > I think you need to be clear on *why* you want to know this
 > information. Once it's clear what you're trying to achieve, it will be
 > easier to say what the options are.

This is what I wrote originally (slightly modified):
  
  A use case here I am thinking of here is in a stack trace or a
  debugger, or a tool which wants to show in great detail, information
  from a code object obtained possibly via a frame object.

I find it kind of sucky to see in a traceback: "" as opposed
to the text (or prefix of the text) of the actual string that was
passed. Or something that has been referred to as a "pseudo-file" like
/usr/lib/python2.5/site-packages/tracer-0.1.0-py2.5.egg/foo/bar.py
when it is really member foo/bar.py of zipped egg
/usr/lib/python2.5/site-packages/tracer-0.1.0-py2.5.egg.

(As a separate issue, it seems that zipimporter file locations inside
setuptools may have a problem.)

Inside a debugger or an IDE, it is conceivable a person might want
loader, and module information, and if the code is part of an archive
file, then member information. (If part of an eval string then, the
eval string.)

 > 
 > It sounds like you're trying to propose a stronger convention, to be
 > enforced in the future. 

Well, I wasn't sure if there was one. But I gather from what you write,
there isn't. :-)

Yes, I would suggest a stronger convention. Or a more up-front
statement that none is desired/forthcoming.

 > (At least, your suggestion of producing stack
 > traces implies that you want stack trace code not to have to deal with
 > the current situation). When PEP 302 was being developed, we were
 > looking at similar issues. That's why I pointed you at get_source() -
 > it was the best we could do with all the various conflicting
 > requirements, and the fact that it's optional is because we had to
 > cater for cases where there simply wasn't a meaningful answer.
 > Frankly, backward compatibility requirements kill a lot of the options
 > here.
 > 
 > Maybe what you want is a *pair* of linked conventions:
 > 
 > - co_filename (or a replacement) returns a (notionally opaque, but
 > in practice a filename for file-based cases) token representing "the
 > file or other object the code came from"

This would be nice.

 > -  xxx.get_source_code(token) is a function (I don't know where,
 > xxx is a placeholder for some "suitable" module) which, given such a
 > token, returns the source, or None if there's no viable concept of
 > "the source".

There always is a viable concept of a source. It's whatever was done
to get the code. For example, if it was via an eval then the source
was the eval function and a string, same for exec. If it's via
database access, well that then and some summary info about what's
known about that. 

 > 
 > Or maybe you want a (possibly separate) a

Re: [Python-Dev] Should there be a way or API for retrieving from a code object a loader method and package file where the code comes from?

[R. Bernstein]

Nick Coghlan writes:
 > 3. Do what a number of standard library APIs (e.g. linecache) that
 > accept filenames do and also accept an optional "module globals"
 > argument. 

Actually, I did this and committed a change (to pydb) before posting
any of these queries. ;-)

If "a number of standard library APIs" are doing the *same* thing,
then shouldn't this exposed as a common routine?

If on the other hand, by "a number" you mean "one" as in linecache --
1 *is* a number too! -- then perhaps the relevant code that is buried
inside the "updatecache" should be exposed on its own.  (As a side
benefit that code can be tested separately too!)

Should I file a feature request for this? 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should there be a way or API for retrieving from a code object a loader method and package file where the code comes from?

[R. Bernstein]

Nick Coghlan writes:
 > R. Bernstein wrote:
 > > Nick Coghlan writes:
 > >  > 3. Do what a number of standard library APIs (e.g. linecache) that
 > >  > accept filenames do and also accept an optional "module globals"
 > >  > argument. 
 > > 
 > > Actually, I did this and committed a change (to pydb) before posting
 > > any of these queries. ;-)
 > > 
 > > If "a number of standard library APIs" are doing the *same* thing,
 > > then shouldn't this exposed as a common routine?
 > > 
 > > If on the other hand, by "a number" you mean "one" as in linecache --
 > > 1 *is* a number too! -- then perhaps the relevant code that is buried
 > > inside the "updatecache" should be exposed on its own.  (As a side
 > > benefit that code can be tested separately too!)
 > > 
 > > Should I file a feature request for this? 
 > 
 > The reason for my slightly odd phrasing is that all of the examples I
 > was originally going to mention (traceback, pdb, doctest, inspect)
 > actually all end up calling linecache to do the heavy lifting.
 > 
 > So it is possible that linecache.getlines() actually *is* the common
 > routine you're looking for 

I never asked about getting the text lines for the source code, no
matter how many times people suggest that as an alternative. :-)

Instead, I was asking about a common way to get information about the
source location for say a frame or traceback object (which might
include package name and type) and suggest that there should be a more
unambiguous way to display this information than seems to be in use at
present.

Part of work to retrieve or displaying that information has to do the
some of the same things that is inside of linecache.updatecache()
*before* it retrieves the lines of the source code (when
possible). And possibly parts of it include parts of what's done in
pieces of the inspect module.

 > - it just needs to be added to the
 > documentation and the __all__ attribute for linecache to be officially
 > supported. Currently, only the single line getline() function is
 > documented and exposed via __all__, but I don't see any reason for that
 > restriction - linecache.getlines() has been there with a stable API
 > since at least Python 2.5.
 > 
 > For cases where you have an appropriate Python object (i.e. a module,
 > function, method, class, traceback, frame or code object) rather than a
 > pseudo-filename, then inspect.getsource() actually jumps through a lot
 > of hoops to try to find the actual source code for that object - in
 > those cases, using the appropriate inspect function is generally a much
 > better idea than trying to interpret __file__ yourself.
 > 
 > Cheers,
 > Nick.

Thanks for the information. I will keep in mind those inspect routines. 

They probably will be a helpful for another problem I had been
wondering about -- how one can determine if there is no code
associated at a given a line and file. (In other words and invalid
location for a debugger line breakpoint, such as because the line
part of a comment or the interior line of a string that spans many
lines)

 > 
 > -- 
 > Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 > ---
 > ___
 > Python-Dev mailing list
 > Python-Dev@python.org
 > http://mail.python.org/mailman/listinfo/python-dev
 > Unsubscribe: 
 > http://mail.python.org/mailman/options/python-dev/rocky%40gnu.org
 > 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3000 Process

[R Morris]

Barry Warsaw wrote:
> 
> On Mon, 2006-03-20 at 13:30 -0500, Terry Reedy wrote:
>> "Guido van Rossum" <[EMAIL PROTECTED]> wrote in message
>> >Barry, if you could create that mailing list, please?
>> 
>> And please mirror it on gmane the same way as this list is.
> 
> Subscription requests have been made.
> -Barry
> 

BTW: It's already archived here and searchable back to the end of August:

http://www.nabble.com/Python---python-dev-f2960.html

Sorry for the intrustion.

Regards,
Rod Morris
Nabble.com

--
View this message in context: 
http://www.nabble.com/Python-3000-Process-t1309231.html#a3506703
Sent from the Python - python-dev forum at Nabble.com.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker Stats

[R. David Murray]

On Mon, 23 Jun 2014 17:52:33 +0200, francis  wrote:
> 
> > Hi,
> > I added a new "stats" page to the bug tracker:
> > http://bugs.python.org/issue?@template=stats
> Thanks Ezio,
> 
> Two questions:
> how hard would be to add (or enhance) a chart with the
> “open issues type enhancement” and “open issues type bug”
> info ?
> 
> In the summaries there is a link to “Issues with patch”,
> means that the ones not listed there are in “needs patch”
> or “new” status?

The stats graphs are based on the data generated for the
weekly issue report.  I have a patched version of that
report that adds the bug/enhancement info.  I'll try to dig
it up this week; someone ping me if I forget :)  It think
the patch will need to be updated based on Ezio's changes.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker Stats

[R. David Murray]

On Tue, 24 Jun 2014 20:43:41 +0200, francis  wrote:
> On 06/24/2014 03:50 AM, Ezio Melotti wrote:
> >>From the first graph you can see that out of the 4500+ open issues,
> > about 2000 have a patch.
> One would like to start with the ones that are bugs ;-) and see some
> status line trying to drop to 0 (is that possible :-) ?)
> 
> > We need more reviewers and committers :)
> more patch writers: yes,
> more patch reviewers: yes,

Anyone can review patches, in case that isn't clear.

> more committers: ?? automate!! :-)

That's a goal of the python-workflow interest group.  Unfortunately
between billable work and GSOC mentoring I haven't had time to do much
there lately.  Our first goal is to make the review step easier to manage
(know which patches really need review, be able to list patches where
community review is thought to be complete) by improving the tracker,
then we'll look at creating the patch gating system Nick has talked
about previously.  Still needs a committer to approve the patch, but it
should increase the throughput considerably.

In the meantime, something that would help would be if people would do
reviews and say on the issue "I think this is commit ready" and have
the issue moved to 'commit review' stage.  Do that a few times where
people who are already triagers/committers agree with you, and you'll
get triage privileges on the tracker.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Another case for frozendict

[Jason R. Coombs]

I repeatedly run into situations where a frozendict would be useful, and every 
time I do, I go searching and find the (unfortunately rejected) PEP-416. I'd 
just like to share another case where having a frozendict in the stdlib would 
be useful to me.

I was interacting with a database and had a list of results from 206 queries:

>>> res = [db.cases.remove({'_id': doc['_id']}) for doc in fives]
>>> len(res)
206

I can see that the results are the same for the first two queries.

>>> res[0]
{'n': 1, 'err': None, 'ok': 1.0}
>>> res[1]
{'n': 1, 'err': None, 'ok': 1.0}

So I'd like to test to see if that's the case, so I try to construct a 'set' on 
the results, which in theory would give me a list of unique results:

>>> set(res)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unhashable type: 'dict'

I can't do that because dict is unhashable. That's reasonable, and if I had a 
frozen dict, I could easily work around this limitation and accomplish what I 
need.

>>> set(map(frozendict, res))
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'frozendict' is not defined

PEP-416 mentions a MappingProxyType, but that's no help.

>>> res_ex = list(map(types.MappingProxyType, res))
>>> set(res_ex)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unhashable type: 'mappingproxy'

I can achieve what I need by constructing a set on the 'items' of the dict.

>>> set(tuple(doc.items()) for doc in res)
{(('n', 1), ('err', None), ('ok', 1.0))}

But that syntax would be nicer if the result had the same representation as the 
input (mapping instead of tuple of pairs). A frozendict would have readily 
enabled the desirable behavior.

Although hashability is mentioned in the PEP under constraints, there are many 
use-cases that fall out of the ability to hash a dict, such as the one 
described above, which are not mentioned at all in use-cases for the PEP.

If there's ever any interest in reviving that PEP, I'm in favor of its 
implementation.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bluetooth 4.0 support in "socket" module

[R. David Murray]

On Mon, 14 Jul 2014 16:42:25 -0400, Terry Reedy  wrote:
> On 7/14/2014 9:57 AM, Tim Tisdall wrote:
> 
> 2 questions not answered yet.
> 
> > Also, is there a method to test changes against all the different *nix
> > variations?
> 
> We have a set of buildbots.
> https://www.python.org/dev/buildbot/
> 
> > Is Bluez the standard across the different *nix variations?
> 
> No idea.

It would be really nice to answer that and the related testing questions.
The socket module has bluetooth support, but there are no tests.
An effort to write some was started at the Bloomberg sprint last month,
but nothing has been posted to the issue yet:

http://bugs.python.org/issue7687

Is Bluetooth 4.0 something different from what the socket module already
has?

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another case for frozendict

[R. David Murray]

On Wed, 16 Jul 2014 03:27:23 +0100, MRAB  wrote:
> Here's another use-case.
> 
> Using the 're' module:
> 
>  >>> import re
>  >>> # Make a regex.
> ... p = re.compile(r'(?P\w+)\s+(?P\w+)')
>  >>>
>  >>> # What are the named groups?
> ... p.groupindex
> {'first': 1, 'second': 2}
>  >>>
>  >>> # Perform a match.
> ... m = p.match('FIRST SECOND')
>  >>> m.groupdict()
> {'first': 'FIRST', 'second': 'SECOND'}
>  >>>
>  >>> # Try modifying the pattern object.
> ... p.groupindex['JUNK'] = 'foobar'
>  >>>
>  >>> # What are the named groups now?
> ... p.groupindex
> {'first': 1, 'second': 2, 'JUNK': 'foobar'}
>  >>>
>  >>> # And the match object?
> ... m.groupdict()
> Traceback (most recent call last):
>File "", line 2, in 
> IndexError: no such group
> 
> It can't find a named group called 'JUNK'.

IMO, preventing someone from shooting themselves in the foot by modifying
something they shouldn't modify according to the API is not a Python
use case ("consenting adults").

> And with a bit more tinkering it's possible to crash Python. (I'll
> leave that as an exercise for the reader! :-))

Preventing a Python program from being able to crash the interpreter,
that's a use case :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another case for frozendict

[R. David Murray]

On Wed, 16 Jul 2014 03:27:23 +0100, MRAB  wrote:
>  >>> # Try modifying the pattern object.
> ... p.groupindex['JUNK'] = 'foobar'
>  >>>
>  >>> # What are the named groups now?
> ... p.groupindex
> {'first': 1, 'second': 2, 'JUNK': 'foobar'}
>  >>>
>  >>> # And the match object?
> ... m.groupdict()
> Traceback (most recent call last):
>File "", line 2, in 
> IndexError: no such group
> 
> It can't find a named group called 'JUNK'.

After I hit send on my previous message, I thought more about your
example.  One of the issues here is that modifying the dict breaks an
invariant of the API.  I have a similar situation in the email module,
and I used the same solution you did in regex: always return a new dict.
It would be nice to be able to return a frozendict instead of having the
overhead of building a new dict on each call.  That by itself might not
be enough reason.  But, if the user wants to use the data in modified form
elsewhere, they would then have to construct a new regular dict out of it,
making the decision to vary the data from what matches the state of the
object it came from an explicit one.  That seems to fit the Python zen
("explicit is better than implicit").

So I'm changing my mind, and do consider this a valid use case, even
absent the crash.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another case for frozendict

[R. David Murray]

On Wed, 16 Jul 2014 14:04:29 -, dw+python-...@hmmz.org wrote:
> On Wed, Jul 16, 2014 at 09:47:59AM -0400, R. David Murray wrote:
> 
> > It would be nice to be able to return a frozendict instead of having the
> > overhead of building a new dict on each call.
> 
> There already is an in-between available both to Python and C:
> PyDictProxy_New() / types.MappingProxyType. It's a one line change in
> each case to return a temporary intermediary, using something like (C):
> Py_INCREF(self->dict)
> return self->dict;
> 
> To
> return PyDictProxy_New(self->dict);
> 
> Or Python:
> return self.dct
> 
> To
> return types.MappingProxyType(self.dct)
> 
> Which is cheaper than a copy, and avoids having to audit every use of
> self->dict to ensure the semantics required for a "frozendict" are
> respected, i.e. no mutation occurs after the dict becomes visible to the
> user, and potentially has __hash__ called.
> 
> 
> > That by itself might not be enough reason.  But, if the user wants to
> > use the data in modified form elsewhere, they would then have to
> > construct a new regular dict out of it, making the decision to vary
> > the data from what matches the state of the object it came from an
> > explicit one.  That seems to fit the Python zen ("explicit is better
> > than implicit").
> > 
> > So I'm changing my mind, and do consider this a valid use case, even
> > absent the crash.
> 
> Avoiding crashes seems a better use for a read-only proxy, rather than a
> hashable immutable type.

Good point.  MappingProxyType wasn't yet exposed when I wrote that email
code.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another case for frozendict

[R. David Murray]

On Wed, 16 Jul 2014 10:10:07 -0700, Devin Jeanpierre  
wrote:
> On Wed, Jul 16, 2014 at 6:37 AM, R. David Murray  
> wrote:
> > IMO, preventing someone from shooting themselves in the foot by modifying
> > something they shouldn't modify according to the API is not a Python
> > use case ("consenting adults").
> 
> Then why have immutable objects at all? Why do you have to put tuples
> and frozensets inside sets, instead of lists and sets? Compare with
> Java, which really is "consenting adults" here -- you can add a
> mutable object to a set, just don't mutate it, or you might not be
> able to find it in the set again.
> 
> Several people seem to act as if the Pythonic way is to not allow for
> any sort of immutable types at all. ISTM people are trying to
> retroactively claim some standard of Pythonicity that never existed.
> Python can and does protect you from shooting yourself in the foot by
> making objects immutable. Or do you have another explanation for the
> proliferation of immutable types, and the inability to add mutable
> types to sets and dicts?
> 
> Using a frozendict to protect and enforce an invariant in the re
> module is entirely reasonable. So is creating a new dict each time.
> The intermediate -- reusing a mutable dict and failing in
> incomprehensible ways if you mutate it, and potentially even crashing
> due to memory safety issues -- is not Pythonic at all.

You'll note I ended up agreeing with you there: when mutation breaks an
invariant of the object it came from, that's an issue.  Which would be
the case if you could use mutable objects as keys.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

[R. David Murray]

On Sun, 10 Aug 2014 13:57:36 +1000, Nick Coghlan  wrote:
> On 10 August 2014 13:20, Antoine Pitrou  wrote:
> > Le 09/08/2014 12:43, Ben Hoyt a écrit :
> >
> >> Just thought I'd share some of my excitement about how fast the all-C
> >> version [1] of os.scandir() is turning out to be.
> >>
> >> Below are the results of my scandir / walk benchmark run with three
> >> different versions. I'm using an SSD, which seems to make it
> >> especially faster than listdir / walk. Note that benchmark results can
> >> vary a lot, depending on operating system, file system, hard drive
> >> type, and the OS's caching state.
> >>
> >> Anyway, os.walk() can be FIFTY times as fast using os.scandir().
> >
> >
> > Very nice results, thank you :-)
> 
> Indeed!
> 
> This may actually motivate me to start working on a redesign of
> walkdir at some point, with scandir and DirEntry objects as the basis.
> My original approach was just too slow to be useful in practice (at
> least when working with trees on the scale of a full Fedora or RHEL
> build hosted on an NFS share).

There is another potentially good place in the stdlib to apply scandir:
iglob.  See issue 22167.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[R. David Murray]

On Sun, 10 Aug 2014 13:12:26 -0700, Glenn Linderman  
wrote:
> On 8/10/2014 1:24 AM, Stephen J. Turnbull wrote:
> > Actually ... if I were a fan of the "".join() idiom, I'd seriously
> > propose 0.sum(numeric_iterable) as the RightThang{tm].  Then we could
> > deprecate "".join(string_iterable) in favor of "".sum(string_iterable)
> > (with the same efficient semantics).
> Actually, there is no need to wait for 0.sum() to propose "".sum... but 
> it is only a spelling change, so no real benefit.
> 
> Thinking about this more, maybe it should be a class function, so that 
> it wouldn't require an instance:
> 
> str.sum( iterable_containing_strings )
> 
> [ or  str.join( iterable_containing_strings ) ]

That's how it used to be spelled in python2.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[R. David Murray]

On Sun, 10 Aug 2014 13:12:26 -0700, Glenn Linderman  
wrote:
> On 8/10/2014 1:24 AM, Stephen J. Turnbull wrote:
> > Actually ... if I were a fan of the "".join() idiom, I'd seriously
> > propose 0.sum(numeric_iterable) as the RightThang{tm].  Then we could
> > deprecate "".join(string_iterable) in favor of "".sum(string_iterable)
> > (with the same efficient semantics).
> Actually, there is no need to wait for 0.sum() to propose "".sum... but 
> it is only a spelling change, so no real benefit.
> 
> Thinking about this more, maybe it should be a class function, so that 
> it wouldn't require an instance:
> 
> str.sum( iterable_containing_strings )
> 
> [ or  str.join( iterable_containing_strings ) ]

Sorry, I mean 'string.join' is how it used to be spelled.  Making it a
class method is indeed slightly different.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python2.7 infinite recursion when loading pickled object

[R. David Murray]

On Mon, 11 Aug 2014 21:43:00 +1000, Chris Angelico  wrote:
> On Mon, Aug 11, 2014 at 9:40 PM, Peter Otten <__pete...@web.de> wrote:
> > Read again. The OP tries to delegate attribute lookup to an (existing)
> > attribute.
> >
> > IMO the root cause of the problem is that pickle looks up __dunder__ methods
> > in the instance rather than the class.
> 
> The recursion comes from the attempted lookup of self.item, when
> __init__ hasn't been called.

Indeed, and this is what the OP missed.  With a class like this, it is
necessary to *make* it pickleable, since the pickle protocol doesn't
call __init__.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[R. David Murray]

On Sat, 23 Aug 2014 00:21:18 +0200, Oleg Broytman  wrote:
>I'm involved in developing and maintaining a few big commercial
> projects that will hardly be ported to Python3. So I'm stuck with
> Python2 for many years and I haven't tried Python3. May be I should try
> a small personal project, but certainly not this year. May be the next
> one...

Yes, you should try it.  Really, it's not the monster you are
constructing in your mind.  The functions that read filenames and return
them as text use surrogate escape to preserve the bytes, and the
functions that accept filenames use surrogate escape to recover those
bytes before passing them back to the OS.  So posix binary filenames
just work, as long as the only thing you depend on is being able to
split and join them on the / character (and possibly the . character)
and otherwise treat the names as black boxes...which is exactly the same
situation you are in in python2.

If you need to read filenames out of a file, you'll need to specify the
surrogate escape error handler so that the bytes will be there to be
recovered when you pass them to the file system functions, but it will
work.

Or, as discussed, you can treat them as binary and use the os level
functions that accept binary input (which are exactly the ones you are
used to using in python2).  This includes os.path.split and
os.path.join, which as noted are the only things you can depend on
working correctly when you don't know the encoding of the filenames.

So, the way to look at this is that python3 is no worse[1] than python2 for
handling posix binary filenames, and also provides additional features
if you *do* know the correct encoding of the filenames.

--David

[1] modulo any remaining API bugs, which is exactly where this thread
started: trying to figure out which APIs need to be able to handle
binary paths and/or surrogate escaped paths so that posix filenames
consistently work as well in python3 as they did in python2).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[R. David Murray]

On Sat, 23 Aug 2014 21:08:29 +1000, Steven D'Aprano  wrote:
> When I started this email, I originally began to say that the actual 
> problem was with byte file names that cannot be decoded into Unicode 
> using the system encoding (typically UTF-8 on Linux systems. But I've 
> actually had difficulty demonstrating that it actually is a problem. I 
> started with a byte sequence which is invalid UTF-8, namely:
> 
> b'ZZ\xdb\xdf\xfa\xff'
> 
> created a file with that name, and then tried listing it with 
> os.listdir. Even in Python 3.1 it worked fine. I was able to list the 
> directory and open the file, so I'm not entirely sure where the problem 
> lies exactly. Can somebody demonstrate the failure mode?

The "failure" happens only when you try to cross from the domain of
posix binary filenames into the domain of text streams (that is, a
stream with a consistent encoding).  If you stick with os interfaces
that handle filenames, Python3 handles posix bytes filenames just fine
(though there may be a few corner-case rough edges yet to be fixed, and
the standard streams was one of them).

The difficultly comes if you try to use a filename that contains
undecodable bytes in a non-os-interface text-context (such as writing it
to a text file that you have declared to be a utf-8 encoding): there you
will get an error...not completely unlike the old "your code works until
your user uses unicode" problem we had in python2, but in this case only
happening in a very narrow set of circumstances involving trying to
translate between one domain (posix binary filenames) and another domain
(io streams with a consistent declared encoding).  This is not a common
operation, but appears to be the one Oleg is concerned about. The old
unicode-blowup errors would happen almost any time someone with a
non-ascii language tried to use a program written by an ascii-only
programmer (which was most of us).

The same problem existed in python2 if your goal was to produce a stream
with a consistent encoding, but now python3 treats that as an error.  If
you really want a stream with an inconsistent encoding, open it as
binary and use the surrogate escape error handler to recover the bytes
in the filenames.  That is, *be explicit* about your intentions.

So yes, we've shifted a burden from those who want non-ascii text to
work consistently to those who wanted inconsistently encoded text to "just
work" (or rather *appear* to "just work").  The number of people who
benefit from the improved text model *greatly* outweighs the number of
people inconvenienced by the new strictness when the domain line (posix
binary filenames to consistently encoded text stream) are crossed.  And
the result is more *valid* programs, and fewer unexpected errors
overall, with no inconvenience unless that domain line is crossed,
and even then the inconvenience is limited to the open call that creates
the binary stream.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[R. David Murray]

On Sat, 23 Aug 2014 19:33:06 +0300, Marko Rauhamaa  wrote:
> "R. David Murray" :
> 
> > The same problem existed in python2 if your goal was to produce a stream
> > with a consistent encoding, but now python3 treats that as an error.
> 
> I have a different interpretation of the situation: as a rule, use byte
> strings in Python3. Text strings are a special corner case for
> applications that have to deal with human languages.

Clearly, then, you are writing unix (or perhaps posix)-only programs.

Also, as has been discussed in this thread previously, any program that
deals with filenames is dealing with human readable languages, even
if posix itself treats the filenames as bytes.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[R. David Murray]

On Tue, 26 Aug 2014 11:25:19 +0900, "Stephen J. Turnbull"  
wrote:
> R. David Murray writes:
> 
>  > Also, as has been discussed in this thread previously, any program that
>  > deals with filenames is dealing with human readable languages, even
>  > if posix itself treats the filenames as bytes.
> 
> That's a bit extreme.  I can name two interesting applications
> offhand: git's object database and the Coda filesystem's containers.

As soon as I hit send I realized there were a few counter examples :)
So, replace "any" with "most".

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[R. David Murray]

On Sun, 24 Aug 2014 13:27:55 +1000, Nick Coghlan  wrote:
> As some examples of where bilingual computing breaks down:
> 
> * My NFS client and server may have different locale settings
> * My FTP client and server may have different locale settings
> * My SSH client and server may have different locale settings
> * I save a file locally and send it to someone with a different locale setting
> * I attempt to access a Windows share from a Linux client (or vice-versa)
> * I clone my POSIX hosted git or Mercurial repository on a Windows client
> * I have to connect my Linux client to a Windows Active Directory
> domain (or vice-versa)
> * I have to interoperate between native code and JVM code
> 
> The entire computing industry is currently struggling with this
> monolingual (ASCII/Extended ASCII/EBCDIC/etc) -> bilingual (locale
> encoding/code pages) -> multilingual (Unicode) transition. It's been
> going on for decades, and it's still going to be quite some time
> before we're done.
> 
> The POSIX world is slowly clawing its way towards a multilingual model
> that actually works: UTF-8
> Windows (including the CLR) and the JVM adopted a different
> multilingual model, but still one that actually works: UTF-16-LE

This kind of puts the "length" of the python2->python3 transition
period in perspective, doesn't it?

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path related questions for Guido

[R. David Murray]

On Thu, 28 Aug 2014 10:15:40 -0700, Glenn Linderman  
wrote:
> On 8/28/2014 12:30 AM, MRAB wrote:
> > On 2014-08-28 05:56, Glenn Linderman wrote:
> >> On 8/27/2014 6:08 PM, Stephen J. Turnbull wrote:
> >>> Glenn Linderman writes:
> >>>   > On 8/26/2014 4:31 AM, MRAB wrote:
> >>>   > > On 2014-08-26 03:11, Stephen J. Turnbull wrote:
> >>>   > >> Nick Coghlan writes:
> >>>
> >>>   > > How about:
> >>>   > >
> >>>   > > replace_surrogate_escapes(s, replacement='\uFFFD')
> >>>   > >
> >>>   > > If you want them removed, just pass an empty string as the
> >>>   > > replacement.
> >>>
> >>> That seems better to me (I had too much C for breakfast, I think).
> >>>
> >>>   > And further, replacement could be a vector of 128 characters, to do
> >>>   > immediate transcoding,
> >>>
> >>> Using what encoding?
> >>
> >> The vector would contain the transcoding. Each lone surrogate would map
> >> to a character in the vector.
> >>
> >>> If you knew that much, why didn't you use
> >>> (write, if necessary) an appropriate codec?  I can't envision this
> >>> being useful.
> >>
> >> If the data format describes its encoding, possibly containing data from
> >> several encodings in various spots, then perhaps it is best read as
> >> binary, and processed as binary until those definitions are found.
> >>
> >> But an alternative would be to read with surrogate escapes, and then
> >> when the encoding is determined, to transcode the data. Previously, a
> >> proposal was made to reverse the surrogate escapes to the original
> >> bytes, and then apply the (now known) appropriate codec. There are not
> >> appropriate codecs that can convert directly from surrogate escapes to
> >> the desired end result. This technique could be used instead, for
> >> single-byte, non-escaped encodings. On the other hand, writing specialty
> >> codecs for the purpose would be more general.
> >>
> > There'll be a surrogate escape if a byte couldn't be decoded, but just
> > because a byte could be decoded, it doesn't mean that it's correct.
> >
> > If you picked the wrong encoding, the other codepoints could be wrong
> > too.
> 
> Aha! Thanks for pointing out the flaw in my reasoning. But that means it 
> is also pretty useless to "replace_surrogate_escapes" at all, because it 
> only cleans out the non-decodable characters, not the incorrectly 
> decoded characters.

Well, replace would still be useful for ASCII+surrogateescape.  Also for
cases where the data stream is *supposed* to be in a given encoding, but
contains undecodable bytes.  Showing the stuff that incorrectly decodes
as whatever it decodes to is generally what you want in that case.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path related questions for Guido

[R. David Murray]

On Thu, 28 Aug 2014 10:54:44 -0700, Glenn Linderman  
wrote:
> On 8/28/2014 10:41 AM, R. David Murray wrote:
> > On Thu, 28 Aug 2014 10:15:40 -0700, Glenn Linderman  
> > wrote:
> >> On 8/28/2014 12:30 AM, MRAB wrote:
> >>> There'll be a surrogate escape if a byte couldn't be decoded, but just
> >>> because a byte could be decoded, it doesn't mean that it's correct.
> >>>
> >>> If you picked the wrong encoding, the other codepoints could be wrong
> >>> too.
> >> Aha! Thanks for pointing out the flaw in my reasoning. But that means it
> >> is also pretty useless to "replace_surrogate_escapes" at all, because it
> >> only cleans out the non-decodable characters, not the incorrectly
> >> decoded characters.
> > Well, replace would still be useful for ASCII+surrogateescape.
> 
> How?

Because there "can't" be any incorrectly decoded bytes in the ASCII part,
so all undecodable bytes turning into 'unrecognized character' glyphs
is useful. "can't" is in quotes because of course if you decode random
binary data as ASCII+surrogate escape you could get a mess just like any
other encoding, so this is really a "more *likely* to be useful" version
of my second point, because "real" ASCII with some junk bytes mixed in
is much more likely to be encountered in the wild than, say, utf-8 with
some junk bytes mixed in (although is probably changing as use of utf-8
becomes more widespread, so this point applies to utf-8 as well).

> > Also for
> > cases where the data stream is *supposed* to be in a given encoding, but
> > contains undecodable bytes.  Showing the stuff that incorrectly decodes
> > as whatever it decodes to is generally what you want in that case.
>
> Sure, people can learn to recognize mojibake for what it is, and maybe 
> even learn to recognize it for what it was intended to be, in limited 
> domains. But suppressing/replacing the surrogates doesn't help with 

Well, it does if the alternative is not being able to display the string
to the user at all.  And yeah, people being able to recognize mojibake
in specific problem domains is what I'm talking about...not perhaps a
great use case, but it is a use case.

> that... would it not be better to replace the surrogates with an escape 
> sequence that shows the original, undecodable, byte value?  Like  \xNN ?

Yeah, that idea has been floated as well, and I think it would indeed be
more useful than the 'unknown character' glyph.  I've also seen fonts
that display the hex code inside a box character when the code point is
unknown, which would be cool...but that can hardly be part of unicode,
can it? :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Fri, 29 Aug 2014 17:11:35 -0400, Donald Stufft  wrote:
> Sorry I was on my phone and didn’t get to fully reply to this.
> > On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg  wrote:
> > 
> > * configuration:
> > 
> >   It would be good to be able to switch this on or off
> >   without having to change the code, e.g. via a command
> >   line switch and environment variable; perhaps even
> >   controlling whether or not to raise an exception or
> >   warning.
> 
> I’m on the fence about this, if someone provides a certificate
> that we can validate against (which can be done without
> touching the code) then the only thing that really can’t be
> “fixed” without touching the code is if someone has a certificate
> that is otherwise invalid (expired, not yet valid, wrong hostname,
> etc). I’d say if I was voting on this particular thing I’d be -0, I’d
> rather it didn’t exist but I wouldn’t cry too much if it did.

Especially if you want an accelerated change, there must be a way to
*easily* get back to the previous behavior, or we are going to catch a
lot of flack.  There may be only 7% of public certs that are problematic,
but I'd be willing to bet you that there are more not-really-public ones
that are critical to day to day operations *somewhere* :)

wget and curl have 'ignore validation' as a command line flag for a reason.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Fri, 29 Aug 2014 18:00:50 -0400, Donald Stufft  wrote:
> 
> On Aug 29, 2014, at 5:42 PM, R. David Murray  wrote:
> > Especially if you want an accelerated change, there must be a way to
> > *easily* get back to the previous behavior, or we are going to catch a
> > lot of flack.  There may be only 7% of public certs that are problematic,
> > but I'd be willing to bet you that there are more not-really-public ones
> > that are critical to day to day operations *somewhere* :)
> > 
> > wget and curl have 'ignore validation' as a command line flag for a reason.
> > 
> 
> Right, that’s why I’m on the fence :)
> 
> On one hand, it’s going to break things for some people, (arguably they are
> already broken, just silently so, but we’ll leave that argument aside) and a
> way to get back the old behavior is good. There are already ways within
> the Python code itself, so that’s covered. From outside of the Python code
> there are ways if the certificate is untrusted but otherwise valid which are
> pretty easy to do. The major “gap” is when you have an actual invalid
> certificate due to expiration or hostname or some other such thing.
> 
> On the other hand Python is not wget/curl and the people who are most
> likely to be the target for a “I can’t change the code but I need to get 
> the
> old behavior back” are people who are likely to not be invoking Python
> itself but using something written in Python which happens to be using
> Python. IOW they might be executing “foobar” not “python -m foobar”.

Right, so an environment variable is better than a command line switch,
for Python.

> Like I said though, I’m personally fine either way so don’t take this as
> being against that particular change!

Ack.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Sat, 30 Aug 2014 14:03:57 +0200, "M.-A. Lemburg"  wrote:
> On 30.08.2014 12:55, Antoine Pitrou wrote:
> > On Sat, 30 Aug 2014 12:46:47 +0200
> > "M.-A. Lemburg"  wrote:
> >>> That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE
> >>> env vars (or, better, by specific settings *inside* the application).
> >>>
> >>> I'm against multiplying environment variables, as it makes it more
> >>> difficult to assess the actual security of a setting. The danger of an
> >>> ill-secure setting is much more severe than with hash randomization.
> >>
> >> You have a point there. So how about just a python run-time switch
> >> and no env var ?
> > 
> > Well, why not, but does it have a value over letting the code properly
> > configure their SSLContext?
> 
> Yes, because when Python changes the default to be validating
> and more secure, application developers will do the same as
> they do now: simply use the defaults ;-)

But neither of those addresses the articulated use case: someone *using*
a program implemented in python that does not itself provide a way to
disable the new default security (because it is *new*).  Only an
environment variable will do that.

Since the environment variable is opt-in, I think the "consenting
adults" argument applies to Alex's demure about "multiple connections".
It could still emit the warnings.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Sun, 31 Aug 2014 03:25:25 +0200, Antoine Pitrou  wrote:
> On Sun, 31 Aug 2014 09:26:30 +1000
> Nick Coghlan  wrote:
> > >>
> > >>   * configuration:
> > >>
> > >> It would be good to be able to switch this on or off
> > >> without having to change the code, e.g. via a command
> > >> line switch and environment variable; perhaps even
> > >> controlling whether or not to raise an exception or
> > >> warning.
> > >>
> > >>   * choice of trusted certificate:
> > >>
> > >> Instead of hard wiring using the system CA roots into
> > >> Python it would be good to just make this default and
> > >> permit the user to point Python to a different set of
> > >> CA roots.
> > >>
> > >> This would enable using self signed certs more easily.
> > >> Since these are often used for tests, demos and education,
> > >> I think it's important to allow having more control of
> > >> the trusted certs.
> > >
> > >
> > > +1 for PEP with above changes.
> > 
> > Ditto from me.
> > 
> > In relation to changing the Python CLI API to offer some of the wget/curl
> > style command line options, I like the idea of providing recipes in the
> > docs for implementing them at the application layer, but postponing making
> > the *default* behaviour configurable that way.
> 
> I'm against any additional environment variables and command-line
> options. It will only complicate and obscure the security parameters of
> certificate validation.
> 
> The existing knobs have already been mentioned in this thread, I won't
> mention them here again.

Do those knobs allow one to instruct urllib to accept an invalid
certificate without changing the program code?

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Sun, 31 Aug 2014 16:45:42 +1000, Nick Coghlan  wrote:
> On 31 August 2014 16:16, Donald Stufft  wrote:
> >
> > On Aug 31, 2014, at 2:09 AM, Nick Coghlan  wrote:
> >
> > At the same time, we need to account for the fact that most existing
> > organisations still trust in perimeter defence for their internal
> > network security, and hence tolerate (or even actively encourage) the
> > use of unsecured connections, or skipping certificate validation,
> > internally. This is actually a really terrible idea, but it's still
> > incredibly common due to the general failure of the technology
> > industry to take usability issues seriously when we design security
> > systems (at least until recently) - doing the wrong "unsafe" thing is
> > genuinely easier than doing things right.
> >
> >
> > Just a quick clarification in order to be a little clearer, this change will
> > (obviously) only effect those who trust perimeter security *and* decided to
> > install an invalid certificate instead of just using HTTP. I'm not saying
> > that
> > this doesn't happen, just being specific (I'm not actually sure why they
> > would
> > install a TLS certificate at all if they are trusting perimeter security,
> > but
> > I'm sure folks do).
> 
> It's the end result when a company wide edict to use HTTPS isn't
> backed up by the necessary documentation and training on how to get a
> properly signed cert from your internal CA (or, even better, when such
> an edict comes down without setting up an internal CA first). Folks
> hit the internet instead, find instructions on creating a self-signed
> cert, install that, and tell their users to ignore the security
> warning and accept the cert. Historically, Python clients have "just
> worked" in environments that required a click-through on the browser
> side, since you had to opt in to checking the certificates properly.
> 
> Self-signed certificates can also be really handy for doing local
> testing - you're not really aiming to authenticate the connection in
> that case, you're just aiming to test that the secure connection
> machinery is all working properly.

Self-signed certificates are not crazy in an internal corporate
environment even when properly playing the defense in depth game.  Once
you've acked the cert the first time, you will be warned if it changes
(like an ssh host key).  Sure, as Nick says the corp could set up an
internal signing authority and make sure everyone has their CA...and
they *should*...but realistically, that is probably relatively rare at
the moment, because it is not particularly easy to accomplish
(distributing the CA everywhere it needs to go is still a Hard Problem,
though it has gotten a lot better).

Given the reality of human nature, even when the documentation
accompanying the HTTPS initiative is good, there will *still* be someone
who hasn't followed the internal rules, yet you really need to talk to
the piece of infrastructure they are maintaining.  At least that one is
short term problem (for some definition of "short" that may be several
months long), but it does exist.

In addition, as has been mentioned before, self-signed certs are often
embedded in *devices* from vendors (I'm looking at you, Cisco).  This is
another area where security conciousness has gotten better (the cert
exists) but isn't good yet (the cert is self-signed and replacing it
isn't trivial when it is even possible; and, because the self-signed cert
happens by defaultit gets left in place).  And in the case of those
embedded certs, the cert can wind up *invalid* (expired) as well as
self-signed.  (This last item is where my concern about being able
to talk to invalid certs comes from.)

And yes, I have encountered all of this in the wild.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Mon, 01 Sep 2014 08:10:58 +1000, Nick Coghlan  wrote:
> On 1 Sep 2014 07:43, "Christian Heimes"  wrote:
> >
> > On 31.08.2014 08:09, Nick Coghlan wrote:
> > > As Antoine says here, I'm also opposed to adding more Python specific
> > > configuration options. However, I think there may be something
> > > worthwhile we can do that's closer to the way browsers work, and has
> > > the significant benefit of being implementable as a PyPI module first
> > > (more on that in a separate reply).
> >
> > I'm on your and Antoine's side and strictly against any additional
> > environment variables or command line arguments. That would make the
> > whole validation process even more complex and harder to understand.
> >
> > There might be a better option to give people and companies the option
> > to tune the SSL module to their needs. Python already have a
> > customization hook for the site module called sitecustomize. How about
> > another module named sslcustomize? Such a module could be used to tune
> > the ssl module to the needs of users, e.g. configure a different default
> > context, add certificates to a default context etc.
> >
> > Companies could install them in a system global directory on their
> > servers. Users could put them in their own user site directory and even
> > each virtual env can have one sslcustomize of its own. It's fully
> > backward compatible, doesn't add any flags and developers have the full
> > power of Python for configuration and customization.
> 
> And means a user specific store (if one became available) could be
> configured there.
> 
> Yes, I think this would address my concerns, especially if combined with a
> clear recipe in the documentation on how to optionally disable cert
> validation at the application layer.
> 
> Assuming sslcustomize was in site-packages rather than the standard library
> directories, you would also be able to use virtual environments with an
> appropriate sslcustomize module to disable cert checking even if the
> application you were running didn't support direct configuration.

It sounds like this would address my concerns as well (I don't really
care *how* it is implemented as long as I don't have to touch the
code of a third party application when I upgrade my python version to
3.5...remember, the context here is backward compatibility concerns).

Does it address the issue of accepting an invalid cert, though?

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR

[R. David Murray]

On Sun, 31 Aug 2014 20:14:50 -0700, Dan Stromberg  wrote:
> On Sun, Aug 31, 2014 at 3:28 PM, Greg Ewing  
> wrote:
> > Victor Stinner wrote:
> >>
> >> As written in the PEP, if you want to be notified of the signal, set a
> >> signal handler which raises an exception.
> >
> > I'm not convinced that this covers all possible use cases.
> > It might be all right if you have control over the signal
> > handler, but what if you don't?
> >
> > I think it's best if the functions in the os module remain
> > thin wrappers that expose the OS functionality as fully and
> > directly as possible. Anything else should be provided
> > by higher-level facilities.
> 
> I'm inclined to agree about keeping the os module thin.  If we were to
> recreate Python today, from scratch, it might make sense to hide this
> by default, but now there's almost certainly code out there that
> depends on the current behavior.
> 
> But I also agree that it's hard to pin down which higher level Python
> library calls are going to be using system calls.  My
> http://stromberg.dnsalias.org/~strombrg/pypty/ program had a problem
> with window resizing for a while (SIGWINCH), and although I use it
> pretty much daily now without problems, I'm still not sure I got 100%
> of the possibilities covered.
> 
> Fortunately, wrapping a system call can be as simple as:
> 
> def retry_on_eintr(function, *args, **kw):
> '''
> Retry a system call until one of the following happens:
> 1) It succeeds
> 2) It errors with something other than EINTR
> '''
> 
> while True:
> try:
> return function(*args, **kw)
> except OSError:
> nothing, extra, nothing2 = sys.exc_info()
> dummy = nothing
> dummy = nothing2
> if extra.errno == errno.EINTR:
> continue
> else:
> raise
> 
> Note that os.read() and os.write() need different handling.

Personally, I really want Python to handle EINTR for me.  And indeed,
that has been what we have been doing for a while now, piece by piece,
bug by bug.  Victor just wants to systematize and document that, and I
think that's a good idea.

We've been consistently treating lack of handling of EINTR as a bug.
If there are *real* cases where that causes a backward compatibility
problem, then we need to know.  But so far, we have gotten zero
complaints about the cases that we have fixed.

--David

PS: I recently switched from using selectors to using a timeout on a
socket because in that particular application I could, and because
reading a socket with a timeout handles EINTR (in recent python
versions), whereas reading a non-blocking socket doesn't.  Under the
hood, a socket with a timeout is a non-blocking socket.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR

[R. David Murray]

On Mon, 01 Sep 2014 08:30:27 +0300, Marko Rauhamaa  wrote:
> "R. David Murray" :
> 
> > PS: I recently switched from using selectors to using a timeout on a
> > socket because in that particular application I could, and because
> > reading a socket with a timeout handles EINTR (in recent python
> > versions), whereas reading a non-blocking socket doesn't. Under the
> > hood, a socket with a timeout is a non-blocking socket.
> 
> Under what circumstances would a nonblocking socket generate an EINTR?

Windows.  Enough said?

The exact error message was:

BlockingIOError on my non-blocking socket: 'a non-blocking socket
operation could not be completed immediately"

Needless to say, I was not expecting this, and was about to tear my
remaining hair out about having to completely restructure the code in
order to be able to handle an EINTR on a read on an FD that I got back
from select as ready, until I realized that the way the code had evolved
the only thing I still needed the select for was the timeout, and that
the EINTR bug in sockets with a timeout had already been fixed (thank
goodness I'm able to use python3.4 for this project).  I got lucky, but
this is clearly a serious problem for writing selectors based code on
Windows.

This should tell you just about everything you need to know about why we
want to fix this problem so that things work cross platform.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR

[R. David Murray]

On Mon, 01 Sep 2014 14:15:52 +0300, Marko Rauhamaa  wrote:
> Charles-François Natali :
> 
> >> Which raises an interesting question: what happens to the os.read()
> >> return value if SIGINT is received?
> >
> > There's no return value, a KeywordInterrupt exception is raised.
> > The PEP wouldn't change this behavior.
> 
> Slightly disconcerting... but I'm sure overriding SIGINT would cure
> that. You don't want to lose data if you want to continue running.
> 
> > As for the general behavior: all programming languages/platforms
> > handle EINTR transparently.
> 
> C doesn't. EINTR is there for a purpose. I sure hope Python won't bury
> it under opaque APIs.
> 
> The two requirements are:
> 
>  * Allow the application to react to signals immediately in the main
>flow.

You don't want to be writing your code in Python then.  In Python
you *never* get to react immediately to signals.  The interpreter
sets a flag and calls the python signal handler later.  Yes, the
call is ASAP, but ASAP is *not* "immediately".

>  * Don't lose information.
> 
> 
> Marko
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Tue, 02 Sep 2014 22:16:18 -, Alex Gaynor  wrote:
> This whole scenario seems to be predicated on a siutation where: You have a
> peer whose certificate you can't change, and you have a piece of code you 
> can't
> change, and you're going to upgrade your Python installation, and you want to
> talk to this peer, and you need to use an encrypted channel, but don't really
> care if it's being MITM'd. It doesn't seem to me that this is reasonably
> Python's responsibility to deal with the fact that you have no ability to
> upgrade any of your infrastructure, except your Python version.

I would say that is an excellent summary, except that you are sometimes
*forced* to use an encrypted channel (the device only opens the https port).
That is, in the real messy world of network devices, accepting an
invalid cert is not a nonsensical operation.  (Of course, what I really
want is to be able to say "I know this cert has invalid fields, accept
it anyway, but warn me if it changes when I connect again".)

It's a good point that the actual number of people who will be hit by
this and can't hack the code in question will be (possibly vanishingly)
small, especially if you only consider python3.  But we've already had
one call for a backport :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Tue, 02 Sep 2014 20:59:54 -0400, Terry Reedy  wrote:
> On 9/2/2014 7:47 PM, Glyph Lefkowitz wrote:
> >
> > On Sep 2, 2014, at 4:28 PM, Nick Coghlan  > > wrote:
> >
> >> On 3 Sep 2014 09:08, "David Reid"  >> > wrote:
> 
> >> > Clearly this change should be backported to Python2.
> >>
> >> Proposing to break backwards compatibility in a maintenance release (...)
> 
> For code that depends on a bug, fixing the bug breaks compatibility 
> between releases without and with the fix.  Nothing new.  We 
> occasionally don't backpart disruptive bugfixes because of that. We also 
> backport some security fixes to otherwise unmaintained versions.
> 
> > As we keep saying, this is not a break in backwards compatibility, it's
> > a bug fix.
> 
> Disagreement on whether changing the default for certificate validation 
> is a 'bugfix' or 'enhancement' (and in the colloquial or python tracker 
> sense) the seems like the crux of the disagreement here.  The case for 
> 'bug', at least in the colloquial meaning, is at least plausible. On the 
> other hand, the tracker meaning of 'behavior' issue (contrary to 
> intention and doc) is more strict in including implementation bugs but 
> not necessarily design bugs.

Up to this point the suggestion has been to change the default only in
3.5, and what we are arguing about is what kind of and how much aid to
give to those for whom 3.5 breaks backward compatibility.  So arguing
about feature versus bug is irrelevant.

The top proposal so far is an sslcustomize.py file that could be used to
either decrease or increase the default security.  This is a much less
handy solution than application options (eg, curl, wget) that allow
disabling security for "this cert" or "this CLI session".  It also is
more prone to unthinking abuse since it is persistent.  So perhaps
it is indeed not worth it.  (That's why I suggested an environment
variable...something you could specify on the command line for a one-off.)

The more I think about it the less I like sslcustomize.  I think I'd
rather do without the ability to turn off checking without modifying
code rather than introduce an intentional vector for a system-wide
decrease in security.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Wed, 03 Sep 2014 16:31:13 +0200, Antoine Pitrou  wrote:
> On Tue, 02 Sep 2014 21:29:16 -0400
> "R. David Murray"  wrote:
> > 
> > The top proposal so far is an sslcustomize.py file that could be used to
> > either decrease or increase the default security.  This is a much less
> > handy solution than application options (eg, curl, wget) that allow
> > disabling security for "this cert" or "this CLI session".  It also is
> > more prone to unthinking abuse since it is persistent.  So perhaps
> > it is indeed not worth it.  (That's why I suggested an environment
> > variable...something you could specify on the command line for a one-off.)
> 
> I'll be fine with not adding any hooks at all, and letting people
> configure their application code correctly :-)

Again, the problem arises when it is not *their* application code, but
a third party tool that hasn't been ported to 3.5.

I'm OK with letting go of this invalid-cert issue myself, given the lack
of negative feedback Twisted got.  I'll just keep my fingers crossed.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Wed, 03 Sep 2014 20:37:38 +0200, Antoine Pitrou  wrote:
> On Wed, 3 Sep 2014 10:54:55 -0700
> Guido van Rossum  wrote:
> > Today (working at Dropbox, a much smaller company!) I don't
> > even remember the last time I had to deal with such a browser
> > complaint -- internal services here all redirect to SSL, and not a
> > browser that can find fault with their certs.
> 
> Good for you. I still sometimes get warnings about expired certificates
> - and sometimes ones that don't exactly match the domain being
> fetched (for example, the certificate wouldn't be valid for that
> specific subdomain - note that CAs often charge a premium for multiple
> subdomains, which why small or non-profit Web sites sometimes skimp on
> them).
> 
> You shouldn't assume that the experience of well-connected people in
> the Silicon Valley is representative of what people over the world
> encounter. Yes, where there's a lot of money and a lot of accumulated
> domain competence, security procedures are updated and followed more
> scrupulously...

Heck, yesterday I got invalid certs from...I think it was roku.com, but
in any case not some obscure little company...the actual cert was an
akamai cert, which means something is configured wrong somewhere.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[R. David Murray]

On Wed, 03 Sep 2014 10:09:36 -0700, Ethan Furman  wrote:
> On 09/03/2014 08:58 AM, R. David Murray wrote:
> >
> > I'm OK with letting go of this invalid-cert issue myself, given the lack
> > of negative feedback Twisted got.  I'll just keep my fingers crossed.
> 
> I apologize if I missed this point, but if we have the source code then it is 
> possible to go in and directly modify the 
> application/utility to be able to talk over https to a router with an invalid 
> certificate?  This is an option when 
> creating the ssl_context?

The immediately preceding paragraph that you didn't quote said that the
context was 3rd party applications, not source code under your control.
Yes, you can (usually) still hack the source, but there are good reasons to
prefer to not do that, unfamiliarity with the codebase being just one of
them.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python docs about comparisons vs. CPython reality

[R. David Murray]

On Sat, 06 Sep 2014 13:34:37 +0200, Jan Kaliszewski  
wrote:
> Are they bugs in the Python docs or just some CPython implementation
> details that are purposely not documented? (but then, again, some of
> the docs seem to be at least not precise...):

You might want to read http://bugs.python.org/issue12067 and contribute
your thoughts there.  There are indeed documentation improvements
under discussion.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Sat, 13 Sep 2014 21:06:21 +1200, Nick Coghlan  wrote:
> On 13 Sep 2014 10:18, "Jeff Allen"  wrote:
> > 4. I think (with Antoine) if Jython supported PEP-383 byte smuggling, it
> would have to do it the same way as CPython, as it is visible. It's not
> impossible (I think), but is messy. Some are strongly against.
> 
> It may be worth trying *without* it (i.e. treat "surrogateescape" as
> equivalent to "strict" initially), and seeing how you go. The main purpose
> of surrogateescape in CPython 3 is to recreate the "arbitrary 8-bit data
> round trips work on POSIX" aspect of CPython 2, which doesn't apply in
> exactly the same way on Jython.
> 
> Compared to the 8-bit vs 16-bit str discrepancy that exists in Python 2,
> "surrogateescape is equivalent to strict" seems like a relatively small
> discrepancy in behaviour.

That would totally break the email package.

It would of course be possible to rewrite email to not use surrogate
escape, but it is a seriously non-trivial undertaking.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Tue, 16 Sep 2014 13:51:23 +1000, Chris Angelico  wrote:
> On Tue, Sep 16, 2014 at 1:34 PM, Stephen J. Turnbull  
> wrote:
> > Jim J. Jewett writes:
> >
> > > In terms of best-effort, it is reasonable to treat the smuggled bytes
> > > as representing a character outside of your unicode repertoire
> >
> > I have to disagree. If you ever end up passing them to something that
> > validates or tries to reencode them without surrogateescape, BOOM!
> > These things are the text equivalent of IEEE NaNs.  If all you know
> > (as in the stdlib) is that you have "generic text", the only fairly
> > safe things to do with them are (1) delete them, (2) substitute an
> > appropriate replacement character for them, (3) pass the text
> > containing them verbatim to other code, and (4) reencode them using
> > the same codec they were read with.
> 
> Don't forget, these are *errors*. These are bytes that cannot be
> correctly decoded. That's not something that has any meaning
> whatsoever in text; so by definition, the only things you can do are
> the four you list there (as long as "codec" means both the choice of
> encoding and the use of the surrogateescape flag). It's like dealing
> with control characters when you need to print something visually,
> except that they have an official solution [1] and surrogateescape is
> unofficial. They're not real text, so you have to deal with them
> somehow.

That isn't the case in the email package.  The smuggled bytes are not
errors[*], they are literally smuggled bytes.  But, as Stephen said, the
only things email does with them are the last three of the four he
listed (if you read (3) as passing it between parts of the email
package): the data comes in as text mixed with binary, and the email
package parses it until it knows what the binary is supposed to be,
turns it back into bytes, and decodes it properly.  The goal is to never
let the smuggled bytes escape out the email APIs as surrogateescape
encoded text; though, in practice, this being consenting-adults Python
and code not being bug free, there are places where people have used the
knowledge of how surrogateescape is used by email to work around both
API and code bugs.

--David

[*] Some of the encoded bytes *are* errors (non-ascii in headers or
undecodable bytes in whatever the CTE/charset is), and in that case
email may just turn them back into error bytes in the output, but only
*some* of the smuggled bytes are actually errors (and none are if the
message is RFC compliant).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Wed, 17 Sep 2014 01:27:44 +1000, Chris Angelico  wrote:
> On Wed, Sep 17, 2014 at 1:00 AM, R. David Murray  
> wrote:
> > That isn't the case in the email package.  The smuggled bytes are not
> > errors[*], they are literally smuggled bytes.
> 
> But they're not characters, which is what Stephen and I were saying -
> and contrary to what Jim said about treating them as characters. At
> best, they represent characters but in some encoding other than the
> one you're using, and you have no idea how many bytes form a character
> or anything. So you can't, for instance, word-wrap the text, because
> you can't know how wide these unknown bytes are, whether they
> represent spaces (wrap points), or newlines, or anything like that.
> You can't treat them as characters, so while you have them in your
> string, you can't treat it as a pure Unicode string - it''s a Unicode
> string with smuggled bytes.

Well, except that I do.  The email header parsing algorithms all work
fine if I treat the surrogate escaped bytes as 'unknown junk' and just
parse based on the valid unicode.  (Unless the header is so garbled that
it can't be parsed, of course, at which point it becomes an invalid
header).

You are right about the wrapping, though.  If a header with invalid
bytes (and in this scenario we *are* talking about errors) needs to
be wrapped, we have to first decode the smuggled bytes and turn it
into an 'unknown-8bit' encoded word before we can wrap the header.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Wed, 17 Sep 2014 04:02:11 +1000, Chris Angelico  wrote:
> On Wed, Sep 17, 2014 at 3:46 AM, R. David Murray  
> wrote:
> >> You can't treat them as characters, so while you have them in your
> >> string, you can't treat it as a pure Unicode string - it''s a Unicode
> >> string with smuggled bytes.
> >
> > Well, except that I do.  The email header parsing algorithms all work
> > fine if I treat the surrogate escaped bytes as 'unknown junk' and just
> > parse based on the valid unicode.  (Unless the header is so garbled that
> > it can't be parsed, of course, at which point it becomes an invalid
> > header).
> 
> Do what, exactly? As I understand you, you treat the unknown bytes as
> completely opaque, not representing any characters at all. Which is
> what I'm saying: those are not characters.

Yes.  I thought you were saying that one could not treat the string with
smuggled bytes as if it were a string.  (It's a string that can't be
encoded unless you use the surrogateescape error handler, but it is
still a string from Python's POV, which is the point of the error
handler).

Or, to put it another way, your implication was that there were no
string operations that could be usefully applied to a string containing
smuggled bytes, but that is not the case.  (I may well have read an
implication that was not there; if so I apologize and you can ignore the
rest of this :)  Basically, we are pretending that the each smuggled
byte is single character for string parsing purposes...but they don't
match any of our parsing constants.  They are all "any character" matches
in the regexes and what have you.  Of course, this only works in
contexts where we can ignore or "carry along" the smuggled bytes as
being components of "arbitrary text" portions of the syntax, and we must
take care to either replace them with valid unicode error glyphs or turn
the string of which the are a part into binary using the same codec and
error handler as we used to ingest them to begin with before emitting
them.  And, of course, we can't *modify* the sections containing the
smuggled bytes, only the syntax-matched sections that surround them; and
things like line wrapping are just an invitation to ugliness and bugs
even if you kept the smuggled bytes sections internally intact.

Finally, to explain what I meant by "except that I do": when I added
back binary support to the email package in Python3, initially I *did
not change the parsing algorithms* in the code.  I just smuggled the
bytes, and then dealt with the encoding/decoding at the API boundaries.
This is the same principle used when dealing with filenames in the API
of Python itself.  *Except* at that boundary, I do not need to worry
about whether a particular string contains smuggled bytes or not.[*]

> If you, instead, represented the header as a list with some str
> elements and some bytes, it would be just as valid (though much harder
> to work with); all your manipulations are done on the str parts, and
> the bytes just tag along for the ride.

Quite a bit harder, which is why I don't do that.

> > You are right about the wrapping, though.  If a header with invalid
> > bytes (and in this scenario we *are* talking about errors) needs to
> > be wrapped, we have to first decode the smuggled bytes and turn it
> > into an 'unknown-8bit' encoded word before we can wrap the header.
> 
> Yeah, and that's going to be a bit messy. If you get 60 characters
> followed by 30 unknown bytes, where do you wrap it? Dare you wrap in
> the middle of the smuggled section?

The point of RFC2047 encoded words is that they are an ASCII
representation of binary data, so once the bytes are "properly" Content
Transfer Encoded (as being in an unknown charset) the string contains no
smuggled bytes and can be wrapped.

--David

[*] I worried a lot that this was re-introducing the bytes/string
problem from python2.  The difference is that if the smuggled bytes
escape from the email API, that's a bug in the email package.  So user
code using the library is *not* in danger of getting mysterious encoding
errors when one day the input is international where before it was all
ASCII.  (Absent bugs in the library.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Wed, 17 Sep 2014 08:57:21 +0900, "Stephen J. Turnbull"  
wrote:
> As long as the Java string manipulation functions don't check for
> surrogates, you should be fine with this representation.  Of course I
> suppose your matching functions (etc) don't check for them either, so
> you will be somewhat vulnerable to bugs due to treating them as
> characters.  But the same is true for CPython, AFAIK.

>From my point of view, the string function laxness is a feature, not a
bug.  But I get what you mean.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

On Wed, 17 Sep 2014 14:42:56 +1000, Steven D'Aprano  wrote:
> On Wed, Sep 17, 2014 at 11:14:15AM +1000, Chris Angelico wrote:
> > On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray  
> > wrote:
> 
> > > Basically, we are pretending that the each smuggled
> > > byte is single character for string parsing purposes...but they don't
> > > match any of our parsing constants.  They are all "any character" matches
> > > in the regexes and what have you.
> > 
> > This is slightly iffy, as you can't be sure that one byte represents
> > one character, but as long as you don't much care about that, it's not
> > going to be an issue.
> 
> This discussion would probably be a lot more easy to follow, with fewer 
> miscommunications, if there were some examples. Here is my example, 
> perhaps someone can tell me if I'm understanding it correctly.
> 
> I want to send an email including the header line:
> 
> 'Subject: “NOBODY expects the Spanish Inquisition!”'
> 
> Note the curly quotes. I've read the manifesto "UTF-8 Everywhere" so I 
> do the right thing and encode it as UTF-8:
> 
> b'Subject: \xe2\x80\x9cNOBODY expects the Spanish Inquisition!\xe2\x80\x9d'

That won't work until email supports RFC 6532.  Until then, you can only
use ascii and encoded words successfully.  So just having the curly
quotes is a buggy enough program.

> but it's not up to Python's email package to throw those invalid bytes 
> out or permantly replace them with something else. Also, we want to work 
> with Unicode strings, not byte strings, so there has to be a way to 
> smuggle those three bytes into Unicode, without ending up with either 
> the replacement bytes:
> 
> # using the 'replace' error handler
> 'Subject: ���NOBODY expects the Spanish Inquisition!”'

What you'll get if you request a text copy of that header is

  'Subject: ���NOBODY expects the Spanish Inquisition!���'

> Am I right so far?
> 
> So the email package uses the surrogate-escape error handler and ends up 
> with this Unicode string:
> 
> 'Subject: \udc9c\udc80\udce2NOBODY expects the Spanish Inquisition!”'

Except that it encodes the closing quote, too :)

> which can be encoded back to the bytes we started with.

Right.  If you serialize the message as bytes, the bytes are recovered
and output when that header is output.

Now, once we support RFC 6532, you will be exactly right, as we will
then have the option of handling utf-8 encoded headers, and we will do
that using the utf-8 codec to ingest headers, and the surrogateescape
error handler to handle exactly the kind of bad data you postulate.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[R. David Murray]

Sorry for the mojibake.  I've not yet gotten around to actually using
the email package to write a smarter replacement for nmh, which is what
I use for email, and I always forget that I need to manually tell nmh
when there non-ascii in the message...

On Wed, 17 Sep 2014 03:02:33 -0400, "R. David Murray"  
wrote:
> On Wed, 17 Sep 2014 14:42:56 +1000, Steven D'Aprano  
> wrote:
> > On Wed, Sep 17, 2014 at 11:14:15AM +1000, Chris Angelico wrote:
> > > On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray  
> > > wrote:
> > 
> > > > Basically, we are pretending that the each smuggled
> > > > byte is single character for string parsing purposes...but they don't
> > > > match any of our parsing constants.  They are all "any character" 
> > > > matches
> > > > in the regexes and what have you.
> > > 
> > > This is slightly iffy, as you can't be sure that one byte represents
> > > one character, but as long as you don't much care about that, it's not
> > > going to be an issue.
> > 
> > This discussion would probably be a lot more easy to follow, with fewer 
> > miscommunications, if there were some examples. Here is my example, 
> > perhaps someone can tell me if I'm understanding it correctly.
> > 
> > I want to send an email including the header line:
> > 
> > 'Subject: “NOBODY expects the Spanish Inquisition!”'
> > 
> > Note the curly quotes. I've read the manifesto "UTF-8 Everywhere" so I 
> > do the right thing and encode it as UTF-8:
> > 
> > b'Subject: \xe2\x80\x9cNOBODY expects the Spanish Inquisition!\xe2\x80\x9d'
> 
> That won't work until email supports RFC 6532.  Until then, you can only
> use ascii and encoded words successfully.  So just having the curly
> quotes is a buggy enough program.
> 
> > but it's not up to Python's email package to throw those invalid bytes 
> > out or permantly replace them with something else. Also, we want to work 
> > with Unicode strings, not byte strings, so there has to be a way to 
> > smuggle those three bytes into Unicode, without ending up with either 
> > the replacement bytes:
> > 
> > # using the 'replace' error handler
> > 'Subject: ���NOBODY expects the Spanish Inquisition!”'
> 
> What you'll get if you request a text copy of that header is
> 
>   'Subject: ���NOBODY expects the Spanish Inquisition!���'
> 
> > Am I right so far?
> > 
> > So the email package uses the surrogate-escape error handler and ends up 
> > with this Unicode string:
> > 
> > 'Subject: \udc9c\udc80\udce2NOBODY expects the Spanish Inquisition!”'
> 
> Except that it encodes the closing quote, too :)
> 
> > which can be encoded back to the bytes we started with.
> 
> Right.  If you serialize the message as bytes, the bytes are recovered
> and output when that header is output.
> 
> Now, once we support RFC 6532, you will be exactly right, as we will
> then have the option of handling utf-8 encoded headers, and we will do
> that using the utf-8 codec to ingest headers, and the surrogateescape
> error handler to handle exactly the kind of bad data you postulate.
> 
> --David
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 394 - Clarification of what "python" command should invoke

[R. David Murray]

On Fri, 19 Sep 2014 10:16:20 -0400, Barry Warsaw  wrote:
> The way I look at it is that "/usr/bin/python" is user interface.
> Distributions are completely free to choose whichever Python they want for
> system scripts, and it's great to see that Fedora is well on their way to
> making Python 3 the default for system scripts.  Debian is also making good
> progress, though we likely won't complete the transition until Jessie+1.

Yep, user interface.  Therefore I think the distribution should let the
user choose :)

Most users will chose python2, because that's what other scripts they
might download and install will expect.  So that should be the default.
But I'll choose python3 (and have, on most of my gentoo linux systems,
which supports having both quite well, for system installed packages).

On the gripping hand, given what Fedora is doing, it is not that hard to
change (or create) the symlinks as an end user to point to the python3
programs even if Fedora doesn't support it directly, so I don't see a
problem with the proposed strategy.

> If the user wants to invoke Python 3, it's not hard to type 'python3' and I
> think that's the message we should be spreading.  That already seems pretty
> ingrained in user habits afaict.

As Donald pointed out, that presents a problem for the future.  But
since there is no good alternative to putting /usr/bin/python3 in the
shebang of scripts that require python3, I don't think there is a
solution and we're just going to have to cross that further bridge when
we come to it.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sysadmin tasks

[R. David Murray]

On Wed, 01 Oct 2014 20:35:29 +1300, Shorya Raj  wrote:
> Just curious, is there any sort of tasklist for any sort of sysadmin sort
> of work surrounding CPython development? There seem to be plenty of tasks
> for the actual coding part, but it would be good to get something up for
> the more systems admin side of things. If there is no one managing that
> side yet, I would be more than happy to start to do so.

We have a sysadmin team (python-infrastructure is the mailing list,
there's also a #python-infra channel on freenode).  I'm sure they will
be happy for additional help!  I don't know that they have a formal task
list, but they might.  (Although I also do sysadmin stuff, I've confined
myself to bugs.python.org for my python sysadmin work...and that isn't
currently handled by the python-infrastructure team).

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] bytes-like objects

[R. David Murray]

Over the past while we've been cleaning up the docs in the area of "how
do we refer to bytes, bytearray, memoryview, etc, etc?" in the APIs that
deal with bytes.  As you may or may not remember, we settled on the term
'bytes-like object', and have changed the docs to (we hope) consistently
use this term, which has a glossary entry most of the references to it
link to.

I just committed (to 3.5) the final changes for issue 16518, the change
to consistently using the term 'bytes-like object' for things that
support the buffer interface.  This means that messages that previously
read:

'sometype' does not support the buffer interface

now read

a bytes-like object is required, not 'sometype'

The 'buffer interface' messages were rather confusing, since you have to
dig to find out what the 'buffer interface' is.  It isn't any sort of
python object, nor is there any object in python3 that has a name
related to it.  'bytes-like object', on the other hand, is fairly
intuitive[*], and if you look it up in the glossary it links to the
explanation of the buffer interface.

We felt it was worth explicitly mentioning this patch on python-dev to point
out to everyone that the docs and error messages now use a consistent
terminology, instead of the mis-mash we had before, which should make it easier
to help people with issues where this comes up.

If there are objections to this change to the messages it is easy enough to
back out, but personally I find the new error messages *much* clearer, and I'm
an experienced dev.

(This work was done by Ezio Melotti, but I committed the final patch as
part of my quest to clear the 'commit review' queue in the tracker.)

--David

[*] the more esoteric cases are not often encountered in regular (as opposed to
NumPy) code, and when they are, the extension to "something that can be
interpreted as a sequence of bytes" is pretty straightforward conceptually.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fixing 2.7.x

[R. David Murray]

On Mon, 06 Oct 2014 21:18:23 +0200, Christian Tismer  
wrote:
> On 06.10.14 20:55, Zachary Ware wrote:
> > On Mon, Oct 6, 2014 at 12:24 PM, Ned Deily  wrote:
> >> 3. security: "fixing issues exploitable by attackers such as crashes,
> >> privilege escalation and, optionally, other issues such as denial of
> >> service attacks. Any other changes are not considered a security risk
> >> and thus not backported to a security branch."
> >>= 3.2.x and 3.3.x
> >
> > 3.1 is still in this category, is it not?  According to PEP 375, it's
> > a few months past due for its last release.
> >
> > http://legacy.python.org/dev/peps/pep-0375/#maintenance-releases
> >
> 
> I don't think that the rules should be implicitly considered
> compatible between the 2.X and 3.X series.
> 
> Python 2.X has a history that extends to X==6, X==5 and
> even X==4, as a really conservative POV with an extent over more
> than 10 years.
> 
> I believe, such a thing does not exist for the Python 3.X series
> at all. My impression is that no 3.X user ever would want to stick
> with any older version.
> 
> Is that true, or am I totally wrong?

I don't think you are *totally* wrong, but I don't think you are really
right, either.  I myself have at least one system (I didn't check them
all) running 3.2 that I have intentionally only done security fixes on
rather than upgrade python3.  It's mostly laziness (given that my distro
provides the security updates), since I don't think there would be any
compatibility problems with the few python3 programs running on it, but
I'm likely not the only one.

So yes, the same rules should apply to python3 as apply to python2,
especially since more distros are about to start shipping python3 as
the system python (Arch has been since 2011).

3.1, however, is most likely a dead issue.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] mUTF-7 support?

[R. David Murray]

On Fri, 10 Oct 2014 01:33:58 +0200, Jesus Cea  wrote:
> On 10/10/14 01:08, Victor Stinner wrote:
> > When you say "IMAP4", do you mean any IMAP4 server? Do you have a list
> > of server vendors known to use the encoding mUTF-7?
> 
> All of them. IMAP4 protocol **REQUIRES** mUTF-7.

[...]

> I am volunteering and can even do the mercurial PUSH myself :-p. That is
> an advantage over some of those new encoding requests :-pp.
> 
> But then yes, I realize that this is a specialized tool (even if IMAP4
> is probably the most popular mail access protocol in the world), we can
> accommodate every use-case and there are tons of mUTF-7 libraries out
> there already.
> 
> So, I am asking :).

I see you are already nosy on issue 5305.  I don't think there is any
question that this is a useful and desired feature for python's imaplib.
If it makes sense to implement it as a codec, there is no reason *not*
to do that.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] mUTF-7 support?

[R. David Murray]

On Fri, 10 Oct 2014 04:28:21 +0200, Antoine Pitrou  wrote:
> On Thu, 9 Oct 2014 19:12:29 -0700
> Dan Stromberg  wrote:
> > On Thu, Oct 9, 2014 at 3:47 PM, Jesus Cea  wrote:
> > > I miss mUTF-7 support (as used to encode IMAP4 mailbox names) in Python,
> > > in the codecs module. As an european with a language with 27 different
> > > letters (instead of english 26), tildes, opening question marks, etc., I
> > > find it very inconvenient.
> > >
> > > This encoding is used basically only in IMAP4, I know. But IMAP4 is an
> > > important protocol and all projects related to it needs mUTF-7 support
> > > if they care about non-english alphabets. Everybody has already an
> > > implementation, waste of effort.
> > 
> > I've been parsing up a huge gmail account with no encoding errors,
> > using CPython 2.x and CPython 3.x.  I'd be surprised if there are no
> > foreign characters in any of the thousands of messages there - but
> > maybe I'm just being very lucky.  I'm not specifying a codec, and I
> > don't see a way of specifying one offhand.
> 
> AFAIU, this is specifically about mailbox names, not messages.

Specifically, it is about what we might better term mailbox
*folders*...that is, not what you would normally think of as the
'mailbox name', which is usually understood to be the thing before the @
in the email address (and can't contain non-ASCII yet...we need RFC 6855
support for that, and I'm not sure *anybody* has that yet).

In this context it is the names you give to folders on the IMAP
server...starting (usually) with INBOX and adding from there.  These
names are used in IMAP commands (ex: the 'select' or 'create' commands),
and IMAP uses mUTF-7 for those.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Internationalized email support (was mUTF-7 support?)

[R. David Murray]

On Fri, 10 Oct 2014 16:16:24 +0900, "Stephen J. Turnbull"  
wrote:
> R. David Murray writes:
> 
>  > in the email address (and can't contain non-ASCII yet...we need RFC 6855
>  > support for that, and I'm not sure *anybody* has that yet).
> 
> If it's an RFC, *somebody* has it *somewhere*. :-)

Ah, it looks like you may be right.  I didn't find any hits in the
first few pages on Google; the closest I got was:

https://groups.google.com/forum/?_escaped_fragment_=topic/mozilla.dev.apps.thunderbird/QBFibOEK5x4#!topic/mozilla.dev.apps.thunderbird/QBFibOEK5x4

I started writing a response that said "if so, it is hidden from
google".  Then I did some deeper digging, and found this:

http://googleblog.blogspot.com/2014/08/a-first-step-toward-more-global-email.html

So my search using the RFC number was just too narrow...since gmail is
IMAP based, Google itself must have *some* sort of RFC 6855 support.
(Unless they've done a hack job of this internationalization support
that only works for their own gmail clients, which given my past
experience working with the gmail IMAP API I would believe...although
I think the IMAP problems I found were mostly bugs rather than
intentional deviations from the standards).

We had a GSOC student (Milan Oberkirch) working on international email
support this past summer, and I believe you asked in the proposal review
phase why working on support for these RFCs was apropos.  At the time I
was kind of hoping Python could kickstart development of
internationalized email by providing tools to build experiments, but now
that Google has announced support, supporting it has become surprisingly
immediately relevant.  (The previously experimental postfix support for
RFC 6531 is targeted for official release in 2.12, which is another big
step forward.)

I'd better get to reviewing the rest of those patches sometime
soonthey include a preliminary patch for RFC 6855.  Unfortunately
the email package itself doesn't handle RFC 6532 email and there's no
proposed patch for that yet.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sad status of Python 3.x buildbots

[R. David Murray]

On Fri, 10 Oct 2014 18:00:06 +0200, Jesus Cea  wrote:
> On 10/10/14 17:56, Chris Angelico wrote:
> > Could I write a little
> > monitor at my end that asks every hour if my buildbots can be seen?
> 
> AFAIK maintainers already get an email if the buildbot vanishes long
> enough. I am more interested in getting an email when my buildbot is
> consistently red because somebody committed something it doesn't like
> two months ago...

I think they are supposed to, but I've never gotten one, not even when
my gentoo buildbots suffered a hardware failure.  A month or so later
Antoine emailed me, and I told him to remove them at least for now,
since I don't currently have replacement hardware.  (I'm hoping to fix
that, but I have to find the time...)

That said, there has never as far as I know been an active hook to email
the maintainer when the buildbot is red.  I don't know if buildbot
supports that or not, so that would be the first thing to check.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sat, 25 Oct 2014 05:45:24 -0700, "Tony Kelman"  wrote:
> As a developer of a (compiled) open-source library or application, wouldn't
> you love to be able to build binaries on Linux for Windows? With some work
> and build system patches, you can. For many projects it's a simple matter of
> ./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
> only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
> Arch. This is possible with a very, very long set of patches, many of which
> have been submitted by Roumen Petrov to the Python bug tracker - see
> http://bugs.python.org/issue17605 and other issues linked therein. Roumen
> has done a huge amount of work, and he and others who consider the MinGW-w64
> compiler important will continue to do so. (Thanks a million Roumen!)

Yes, I for one appreciate Roumen's work, even though I'm not currently
in a position to review the patches.

> > I could step in as maintainer for Cygwin and builds based on GCC using
> > mingw* APIs.
> >
> > Regards,
> > Roumen Petrov
> 
> A maintainer has volunteered. Others will help. Can any core developers
> please begin reviewing some of his patches? Even if just to say they need
> to be rebased. The lack of responses on the bug tracker is disheartening
> from an outside perspective. The pile of patches accumulating in external
> MinGW packaging projects is tantamount to a fork of CPython. It won't go
> away since there are dedicated packagers working to keep their MinGW-w64
> builds functional, even in the ad-hoc current state. The patches continue
> piling up, making it more difficult for everyone - except for the core
> Python developers if they continue to ignore the problem. Bring the people
> working on these patches into the fold as contributors. Review the patches.
> It would make Python as a language and a community even more diverse and
> welcoming.

IIUC, our general policy for bringing new platforms into core support,
as opposed to continuing to be a separately maintained "set of patches",
is that there be a "big enough" community of interest (I don't remember
the definition of "big enough") and that there be both committed
maintainers *and* at least one buildbot.

Being able to build windows packages on linux is a compelling argument,
but that only applies to building extensions, not the interpreter.

I would recommend starting with any patches that help MinGW that are not
MinGW specific but instead generally improve the build system and cross
compilation.  There have been a number of such issues opened and
improvements made beyond the MinGW related ones, some coming from
Debian, some coming from the Android community.

So  target those first.  My suggestion would be to pick a patch that is
believed to be commit ready, and come to #python-dev on freenode and
gently bug us until it gets committed.  Then pick the next one, and
repeat.  Working from simplest to more complex would also probably be a
good strategy :)

>From there I'd move on to patches that support using MinGW for building
extensions.  There will probably be useful to also get engaged with the
packaging folks.  And, at this point, we would NEED A BUILDBOT.  That
is, a machine that has whatever tools are required installed such that
tests added to the test suite to test MinGW support in distutils would
run, so we can be sure we don't break anything when making other
changes.

For compiling CPython itself with MinGW, we'd first need to develop a
consensus that we want to support it in core.  I'd say get building
extensions working first, and then make that argument.

By the time a bunch of patches get committed, we should be ready (read:
eager :) to promote someone to do it themselves.  Even if we never
decide to support compiling CPython itself with MinGW, I would hope that
getting it to work for extensions would greatly reduce the number of
patches needed to be maintained outside the tree in order to do so.  And
once at least one MinGW advocate is part of the core team, advocacy
becomes easier :)

--David

PS: one meta comment about the MinGW issues: I scanned a few from the
linked bug, and while the issues split the patches out, which is a great
start, there is no discussion on the issues for the individual patches
to give background and motivation and an explanation of what the patch
is trying to accomplish.  But you probably don't want to waste time on
improving ones that apply *only* to compiling CPython itself unless we
get consensus that we want to support that.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sat, 25 Oct 2014 21:10:23 +0100, Ray Donnelly  
wrote:
> On Sat, Oct 25, 2014 at 6:13 PM, Steve Dower  
> wrote:
> > (Apologies for the short reply, posting from my phone.)
> >
> > "MSVC can continue
> > to be the default compiler used for Python on Windows, none of Roumen's
> > patches change that. They would merely open up the choice for packagers and
> > users to build CPython (and extension modules, thanks to separate patches)
> > with alternate compilers, in cross-compilation or otherwise."
> >
> > Building CPython for Windows is not something that needs solving. The
> > culture on Windows is to redistribute binaries, not source, and both the
> > core team and a number of redistributors have this figured out (and it will
> > only become easier with VC14 and Python 3.5).
> 
> This is the second time you've used the vacuous "culture on Windows"
> argument, now with an added appeal to (vague) authority. That may be
> your opinion and that of some others, but there's a large number of
> people who don't care for using non-Free tools. IMHO building CPython
> on Windows using Open Source toolchains is very much something that
> needs merging upstream and supporting by default. What is it that you
> are afraid of if CPython can be compiled out of the box using
> mingw/MinGW-w64? Why are you fighting so hard against having option.
> If CPython wants to truly call itself an Open Source project then I
> consider being able to compile and cross-compile it with capable Open
> Source toolchains on all major platforms a requirement.

You are doing yourself a disservice by this last statement.  There
really can't be any question that Python is an open source project,
so insinuating that the CPython community is "doing something wrong"
is not going to win you friends and helpers.

A better approach would be to acknowledge that what we are currently
doing works well for supporting Windows (especially since we actually
have some engagement from Microsoft that is *getting some problems
fixed* in ways that help make things more open).

And then say, "wouldn't it be *really cool* if we could also build
CPython using an open source toolchain on Windows out of the box?".  You
might not get instant agreement on that (well, clearly you won't), but
it'd be much more likely you'd start garnering support.

Assume that people are well intentioned, and convince them your
suggestions will make things *even better* using positive arguments.
You might not succeed, but you'll have a much better chance.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sun, 26 Oct 2014 00:19:44 +0200, Antoine Pitrou  wrote:
> On Sun, 26 Oct 2014 09:06:36 +1100
> Chris Angelico  wrote:
> > On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou  wrote:
> > > How do you know this isn't a problem, since you haven't *tested* with
> > > MSVC?
> > > Why on Earth would you want to test your PEP work with an unsupported
> > > Windows compiler and runtime, rather than with the officially supported
> > > compiler and runtime?
> > 
> > This discussion revolved around supporting MinGW in addition to MSVC.
> > If it had been supported when I was doing that, I could have spun
> > myself up a Windows build and tested it.
> 
> My point is that your "Windows build" would not have the same behaviour
> as a MSVC-produced Windows build, and so testing it with it would not
> certify that your code would actually be compatible with genuine
> MSVC builds of CPython, which we will not stop supporting.

While true, I don't think that matters for Chris' point.  Given only the
ability to build with the MSVC toolchain, his code (which might even be
pure python for the purposes of this discussion) would not get tested on
Windows until committed and run by the buildbots.  If he could build
CPython using MinGW, he would, and would test his code on Windows.  Even
if there are C components and MSVC/MinGW compatibility issues are
revealed when the buildbots eventually run the code, still the number of
bugs present would probably be lower if he had tested it on Windows
than if he hadn't.

I know I for one do not generally test patches on Windows because I
haven't taken the time to learn how to build CPython on it.  Sure, I
could test pure python changes by applying patches to an installed
Python, but that's an ongoing pain and I'd rather learn to build CPython
on Windows and get to use the normal hg tools.

If I could use a more linux-like toolchain to build CPython on windows,
I would doubtless do much more testing on windows for stuff where I
think windows might behave differently (and I might look at more Windows
bugs...though frankly there are plenty of bugs for me to look at without
looking at Windows bugs).

This is not necessarily a compelling argument for MinGW support.
However, it *is* a valid argument, IMO.

Note: it can be made even less compelling by making it a lot easier to
build CPython on Windows without having an MSVC license (which I think
means not using the GUI, for which I say *yay* :).  I think Zach Ware
has been working on improving the Windows build process, and I keep
meaning to give it a try...

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Sun, 26 Oct 2014 06:12:45 -0700, "Tony Kelman"  wrote:
> Steve Dower:
> > Building CPython for Windows is not something that needs solving.
> 
> Not in your opinion, but numerous packagers of MinGW-based native or
> cross-compiled package sets would love to include Python. The fact
> that they currently can't, without many patches, is a problem.

If this includes (or would likely include) a significant portion of the
Scientific Computing community, I would think that would be a compelling
use case.  We'd need to be educated more about the reasons why this
approach works better than remaining compatible with MSVC CPython so we
could evaluate the risks and reward intelligently.  (I wonder..."things
are going to fragment anyway if you (cpython) don't do anything" might
be an argument, if true...but wouldn't make the consequences any
easier to deal with :(

But as has been discussed, it seems better to focus first on fixing the
issues on which we are all in agreement (building extensions with MinGW).

> R. David Murray:
> > And, at this point, we would NEED A BUILDBOT.  That is, a machine that
> > has whatever tools are required installed such that tests added to the
> > test suite to test MinGW support in distutils would run, so we can be
> > sure we don't break anything when making other changes.
> 
> That's not too hard. I've done this for other projects. AppVeyor works if
> your build is short enough, and I've done cross-compilation from Travis
> CI for other projects. Or Jenkins, or a Vagrant VM. I don't know PSF's
> infrastructure, but I can offer guidance if it would help.

When I say "we need a buildbot", what I mean is that we need someone
willing to donate the resources and the *time and expertise* to setting
up and maintaining something that integrates with our existing buildbot
setup.  You set up a buildbot slave, request an id and password from
Antoine, keep the thing running, and respond in a timely fashion to
requests for help resolving issues that arise on the buildbot (both
buildbot issues and help-me-diagnose-this-failure issues).  After the
initial setup the load isn't generally heavy (I haven't had to do
anything with the OSX buildbot running on the machine in my dinning room
for months and months now, for example).

So your guidance would have to go to someone who was volunteering to
take on this task...there isn't anyone on the existing core team who
would have time to do it (if I'm wrong, I'm sure someone will speak up).
On the other hand, you don't have to be a committer to run a buildbot,
and there *are* people on the core-mentorship list who have expressed
interest in helping out with our automated testing infrastructure,
including (if I understand correctly) adding some level of integration
to other CI systems (which might just be messages to the IRC
channel)[*].  So that could be a fruitful avenue to explore.

--David

[*] This is an area in which I have an interest, but it hasn't gotten
high enough on my todo list yet for me to figure out exactly what the
current state of things is so I can help it along.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Wed, 29 Oct 2014 10:22:14 -0400, Tres Seaver  wrote:
> On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
> 
> > most developers on Windows do have access to Microsoft tool
> 
> I assume you mean python-dev folks who work on Windows:  it is certainly
> not true for the vast majority of develoeprs who use Python on Windows,
> who don't have the toolchain build their own C extensions.

I'm pretty sure he meant "most people who develop software for Windows",
even though that's not how he phrased it.  But this does not include, as
you point out, people who develop Python software that *also* works on
Windows.

If you are writing code targeted for Windows, I think you are very
likely to have an MSDN subscription of some sort if your package includes C
code.  I'm sure it's not 100%, though.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Thu, 30 Oct 2014 01:09:45 +1000, Nick Coghlan  wrote:
> (Paul Moore already covered most of this, but I'll go into a bit more
> detail in a couple of areas)
> 
> On 29 October 2014 00:46, Tony Kelman  wrote:
> > Stephen J. Turnbull:
> >> It should be evident by now that our belief is that the large majority
> >> of Windows users is well-served by the current model
> >
> >
> > This is not the case at all in the scientific community. NumPy and SciPy
> > put in a lot of extra work to come up with something that is compatible
> > with the MSVC build of CPython because they have to, not because they're
> > "happy to" jump through the necessary hoops.
> 
> Lots of folks are happy with POSIX emulation layers on Windows, as
> they're OK with "basically works" rather than "works like any other
> native application". "Basically works" isn't sufficient for many
> Python-on-Windows use cases though, so the core ABI is a platform
> native one, rather than a POSIX emulation.

Since some of the context here is scientific use of Python, it might be
a useful bit of perspective to know that, while there are doubtless many
scientists using windows and using the windows native interfaces
happily, the Software Carpentry bootcamps that aim to give scientists
the basic framework for making better use of computers and software and
programming have as one foundation the bash shell, taught on Windows via
git-bash.  That is, the common toolset being taught to scientists (by
Software Carpentry) is the posix one, even on Windows.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[R. David Murray]

On Wed, 29 Oct 2014 23:33:06 -0400, Terry Reedy  wrote:
> On 10/29/2014 4:05 PM, Paul Moore wrote:
> > On 29 October 2014 15:31, Nathaniel Smith  wrote:
> >>> You can use Express editions of Visual Studio.
> >>
> >> IIUC, the express edition compilers are 32-bit only, and what you actually
> >> want are the "SDK compilers":
> >> https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
> >>
> >> These are freely downloadable by anyone, no msdn subscription required, but
> >> only if you know where to find them!
> >>
> >> AFAICT the main obstacle to using MSVC to build python extensions (assuming
> >> it can handle your code at all) is not anything technical, but rather that
> >> there's no clear and correct tutorial on how to do it, and lots of 
> >> confusion
> >> and misinformation circulating.
> >
> > Would it help if I wrote a document explaining how to set up the MS
> > compilers (free and paid for) to allow building of Python extensions?
> 
> > There are a few provisos:
> >
> > 1. A lot of it will be pretty trivial: "If you have Vistal Studio
> > (full edition), install it. Done."
> > 2. It will be out of date very fast as the situation for Python 3.5+
> > will be trivial across the board.
> > 3. I don't have anywhere particularly authoritative to host it (maybe
> > the Python Wiki?) and it could easily get lost in the huge swamp of
> > variously outdated, over-complicated, or otherwise alternative
> > documents available. Ideally I'd like someone to suggest an "official"
> > location I could use.
> 
> There is already a section in the devguide for building Python itself, 
> with mostly the same info, except it may not be up to date.
> 
> > I don't want to do this if it won't be useful, as it'll take me a bit
> > of effort to confirm the process for the only non-trivial scenario
> > (64-bit Python 3.3/3.4 with free tools). But if people think it would
> > help, that's fine, I volunteer.
> 
> The devguide needs to be kept up to date.  If you open a tracker issue, 
> put me as nosy to review and commit.

The devguide is about building python itself.  Paul is talking about
building extensions.  That should go in the packaging docs.  I don't
*think* Paul is volunteering to explain how to build python itself,
that's pretty much our job :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The role of NotImplemented: What is it for and when should it be used?

[R. David Murray]

on in my code
> [2] usually a TypeError, stating either that the operation is not supported, 
> or the types are unorderable
> [3] test script at the end
> [4] https://bugs.python.org/issue22766 [returning NotImplemented was rejected]
> 
> -- 8< 
> 
> from collections import Counter, defaultdict, deque, OrderedDict
> from fractions import Fraction
> from decimal import Decimal
> from enum import Enum
> import operator
> import sys
> 
> py_ver = sys.version_info[:2]
> 
> types = (
>  bytes, bytearray, str, dict, list, tuple,
>  Enum, Counter, defaultdict, deque, OrderedDict,
>  )
> numeric_types = int, float, Decimal, Fraction
> 
> operators = (
>  '__add__', '__and__', '__floordiv__',
>  '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__',
>  '__imod__', '__imul__', '__ior__', '__ipow__',
>  '__irshift__', '__isub__', '__itruediv__', '__ixor__',
>  '__lshift__', '__mod__', '__mul__',
>  '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__',
>  '__xor__',
>  )
> 
> if py_ver >= (3, 0):
>  operators += ('__gt__',  '__ge__', '__le__','__lt__')
> 
> ordered_reflections = {
>  '__le__': '__ge__',
>  '__lt__': '__gt__',
>  '__ge__': '__le__',
>  '__gt__': '__lt__',
>  }
> 
> 
> # helpers
> 
> class SomeOtherClass:
>  """"
>  used to test behavior when a different type is passed in to the
>  special methods
>  """
>  def __repr__(self):
>  return 'SomeOtherClass'
> some_other_class = SomeOtherClass()
> 
> class MainClassHandled(Exception):
>  """
>  called by base class if both operands are of type base class
>  """
> 
> class SubClassCalled(Exception):
>  """
>  called by reflected operations for testing
>  """
> 
> def create_control(test_op):
>  def _any(self, other):
>  if not type(other) is self.__class__:
>  return NotImplemented
>  raise MainClassHandled
>  class Control:
>  "returns NotImplemented when other object is not supported"
>  _any.__name__ = op
>  setattr(Control, test_op, _any)
>  return Control()
> 
> def create_subtype(test_op, base_class=object):
>  def _any(*a):
>  global subclass_called
>  subclass_called = True
>  raise SubClassCalled
>  class subtype(base_class):
>  __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any
>  __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any
>  __and__ = __xor__ = __or__ = _any
>  __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ = _any
>  __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ = _any
>  __rand__ = __rxor__ = __ror__ = _any
>  __le__ = __lt__ = __gt__ = __ge__ = _any
>  if issubclass(subtype, (bytes, bytearray)):
>  value = b'hello'
>  elif issubclass(subtype, str):
>  value = 'goodbye'
>  elif issubclass(subtype, (list, tuple)):
>  value = (1, 2, 3)
>  elif issubclass(subtype, (int, float, Decimal, Fraction)):
>  value = 42
>  else:
>  # ignore value
>  return subtype()
>  return subtype(value)
> 
> 
> # test exceptions
> 
> # control against some other class
> print('testing control...\n')
> errors = False
> for op in operators:
>  control = create_control(op)
>      op = getattr(operator, op)
>  try:
>  op(control, some_other_class)
>  except TypeError:
>  # the end result of no method existing, or each method called 
> returning
>  # NotImplemented because it does not know how to perform the 
> requested
>  # operation between the two types
>  pass
>  except Exception as exc:
>  errors = True
>  print('%s(%s()) -- Exception <%s> raised instead of TypeError' %
>  (op.__name__, test_type.__name__, exc))
>  else:
>  errors = True
>  print('Control -- TypeError not 

Re: [Python-Dev] The role of NotImplemented: What is it for and when should it be used?

[R. David Murray]

On Mon, 03 Nov 2014 15:05:31 +, Brett Cannon  wrote:
> On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman  wrote:
> 
> > Just to be clear, this is about NotImplemented, not NotImplementedError.
> >
> > tl;dr  When a binary operation fails, should an exception be raised or
> > NotImplemented returned?
> >
> 
> The docs for NotImplemented suggest it's only for rich comparison methods
> and not all binary operators:
> https://docs.python.org/3/library/constants.html#NotImplemented . But then
> had I not read that I would have said all binary operator methods should
> return NotImplemented when the types are incompatible.

Ethan opened an issue and then changed those docs, but I now believe
that the docs should be changed back (see the discussion in issue
22766).

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Real-world use of Counter

[R. David Murray]

On Wed, 05 Nov 2014 11:13:37 -0800, Ethan Furman  wrote:
> On 11/05/2014 10:56 AM, Raymond Hettinger wrote:
> > The in-place operations on counters are duck-typed.  They are intended (by 
> > design) to work with ANY type that has an
> > items() method.   The exception raised if doesn't have on is an 
> > AttributeError saying that the operand needs to have an
> > items() method.
> 
> It would still be duck-typed with a `hasattr` call on the second operand 
> checking for the necessary method, a TypeError 
> could just as easily state the problem is a missing `items()` method, and 
> then those three [*] in-place methods would 
> raise the same error as the 20(?) other Counter methods under similar 
> conditions.

The technique of duck typing as used in Python is you just let errors
bubble up when the input doesn't work.  Adding special case checks
should in general only be done if the resulting error message is
otherwise really confusing, which is not the case here.

As I said on the issue, there is no reason I can see to add extra code
just to turn an AttributeError into a TypeError.  The AttributeError
works just fine in letting you know your input type didn't work.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Problem displaying the warning symbol

[R. David Murray]

On Sun, 16 Nov 2014 11:23:41 +0100, Stefano Borini 
 wrote:
> I report a finding (bug?) about the warning unicode symbol, as reported here
> 
> http://stackoverflow.com/questions/26919799/cannot-make-warning-sign-visible-on-osx-terminal

Bugs should be reported to bugs.python.org.  They just get lost here.

I'm doubtful this is a Python problem though.  More likely it is an
issue with the curses library on OSX.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[R. David Murray]

On Sun, 23 Nov 2014 00:59:42 +1000, Nick Coghlan  wrote:
> OK, different question. Has anyone here actually even *read* the workflow
> PEPs I wrote? They were on the agenda for the language summit, but got
> bumped due to lack of time (which I'm still annoyed about, given the
> comparatively inconsequential things that chewed up a whole lot of the day).

I have (but I'm core).  It's been a while, though, I need to review it.

> I've only had a couple of folks from outside the core dev community express
> interest in them. Personally, the lack of online editing support annoys me
> immensely whenever I need to work on PEPs or the devguide. I also think

Eh, I hate online editing ;).  But I understand its utility.

[...]

> My preferred answer remains setting up a srlf-hosted forge.python.org, but
> I've seen little evidence we have the capacity to deploy & maintain such a
> service effectively, given the relative lack of interest shown in the idea
> by almost everyone I've spoken to about it. Any progress has only come with
> a lot of pushing from me, and I just don't have the personal bandwidth to
> sustain that at this point. That's why the related PEPs were deferred, and
> the only responses I've received regarding potentially taking them over
> have come from folks outside the core development community, which really
> doesn't help very much in removing my availability as a bottleneck in the
> workflow improvement process.
> 
> If nobody wants to maintain a self-hosted forge, or even enable the folks
> that have expressed interest in setting it up & maintaining it, then the
> right answer is "don't do it" - we should use a commercial service instead.

I have an interest in this, but like you lack bandwidth.  I have no
employer that I could petition for 50% open source time, so any time I
spend on it is time I'm not billing...so my available time comes and
goes depending on my client situation, and I'm also one of the few
putting any time in on tracker maintenance (not that I have much lately,
Ezio has done most of it) and I've still got the email docs to finish up
as well as a list of bugs...yeah, bandwidth is a problem.

But maybe I can take over coordination of the volunteers that want to
work on the forge.  We've already got the mailing list set up.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My thinking about the development process

[R. David Murray]

On Fri, 05 Dec 2014 15:17:35 -0700, Eric Snow  
wrote:
> On Fri, Dec 5, 2014 at 1:04 PM, Brett Cannon  wrote:
> > We don't exactly have a ton of people
> > constantly going "I'm so bored because everything for Python's development
> > infrastructure gets sorted so quickly!" A perfect example is that R. David
> > Murray came up with a nice update for our workflow after PyCon but then ran
> > out of time after mostly defining it and nothing ever became of it (maybe we
> > can rectify that at PyCon?). Eric Snow has pointed out how he has written
> > similar code for pulling PRs from I think GitHub to another code review
> > tool, but that doesn't magically make it work in our infrastructure or get
> > someone to write it and help maintain it (no offense, Eric).
> 
> None taken.  I was thinking the same thing when I wrote that. :)
> 
> >
> > IOW our infrastructure can do anything, but it can't run on hopes and
> > dreams. Commitments from many people to making this happen by a certain
> > deadline will be needed so as to not allow it to drag on forever. People
> > would also have to commit to continued maintenance to make this viable
> > long-term.

The biggest blocker to my actually working the proposal I made was that
people wanted to see it in action first, which means I needed to spin up
a test instance of the tracker and do the work there.  That barrier to
getting started was enough to keep me from getting started...even though
the barrier isn't *that* high (I've done it before, and it is easier now
than it was when I first did it), it is still a *lot* higher than
checking out CPython and working on a patch.

That's probably the biggest issue with *anyone* contributing to tracker
maintenance, and if we could solve that, I think we could get more
people interested in helping maintain it.  We need the equivalent of
dev-in-a-box for setting up for testing proposed changes to
bugs.python.org, but including some standard way to get it deployed so
others can look at a live system running the change in order to review
the patch.

Maybe our infrastructure folks will have a thought or two about this?
I'm willing to put some work into this if we can figure out what
direction to head in.  It could well be tied in to moving
bugs.python.org in with the rest of our infrastructure, something I know
Donald has been noodling with off and on; and I'm willing to help with
that as well.

It sounds like being able to propose and test changes to our Roundup
instance (and test other services talking to Roundup, before deploying
them for real) is going to be critical to improving our workflow no
matter what other decisions are made, so we need to make it easier to
do.

In other words, it seems like the key to improving the productivity of
our CPython patch workflow is to improve the productivity of the patch
workflow for our key workflow resource, bugs.python.org.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker test instances (was: My thinking about the development process)

[R. David Murray]

On Sat, 06 Dec 2014 15:21:46 +, Brett Cannon  wrote:
> On Sat Dec 06 2014 at 10:07:50 AM Donald Stufft  wrote:
> > On Dec 6, 2014, at 9:11 AM, Brett Cannon  wrote:
> >
> >> On Fri Dec 05 2014 at 8:31:27 PM R. David Murray 
> >> wrote:
> >>> That's probably the biggest issue with *anyone* contributing to tracker
> >>> maintenance, and if we could solve that, I think we could get more
> >>> people interested in helping maintain it.  We need the equivalent of
> >>> dev-in-a-box for setting up for testing proposed changes to
> >>> bugs.python.org, but including some standard way to get it deployed so
> >>> others can look at a live system running the change in order to review
> >>> the patch.
> >>
> >> Maybe it's just me and all the Docker/Rocket hoopla that's occurred over
> >> the past week, but this just screams "container" to me which would make
> >> getting a test instance set up dead simple.
> >
> > Heh, one of my thoughts on deploying the bug tracker into production was
> > via a container, especially since we have multiple instances of it. I got
> > side tracked on getting the rest of the infrastructure readier for a web
> > application and some improvements there as well as getting a big postgresql
> > database cluster set up (2x 15GB RAM servers running in Primary/Replica
> > mode). The downside of course to this is that afaik Docker is a lot harder
> > to use on Windows and to some degree OS X than linux. However if the
> > tracker could be deployed as a docker image that would make the
> > infrastructure side a ton easier. I also have control over the python/
> > organization on Docker Hub too for whatever uses we have for it.
> >
> 
> I think it's something worth thinking about, but like you I don't know if
> the containers work on OS X or Windows (I don't work with containers
> personally).

(Had to fix the quoting there, somebody's email program got it wrong.)

For the tracker, being unable to run a test instance on Windows would
likely not be a severe limitation.  Given how few Windows people we get
making contributions to CPython, I'd really rather encourage them to
work there, rather than on the tracker.  OS/X is a bit more problematic,
but it sounds like it is also a bit more doable.

On the other hand, what's the overhead on setting up to use Docker?  If
that task is non-trivial, we're back to having a higher barrier to
entry than running a dev-in-a-box script...

Note also in thinking about setting up a test tracker instance we have
an additional concern: it requires postgres, and needs either a copy of
the full data set (which includes account data/passwords which would
need to be creatively sanitized) or a fairly large test data set.  I'd
prefer a sanitized copy of the real data.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My thinking about the development process

[R. David Murray]

On Mon, 08 Dec 2014 12:27:23 -0800, "Jim J. Jewett"  
wrote:
> Brett Cannon wrote:
> > 4. Contributor creates account on bugs.python.org and signs the
> >   [contributor agreement](https://www.python.org/psf/contrib/contrib-form/)
> 
> Is there an expiration on such forms?  If there doesn't need to be
> (and one form is good for multiple tickets), is there an objection
> (besides "not done yet") to making "signed the form" part of the bug
> reporter account, and required to submit to the CI process?  (An "I
> can't sign yet, bug me later" option would allow the current workflow
> without the "this isn't technically a patch" workaround for "small enough"
> patches from those with slow-moving employers.)

No expiration.  Whether or not we have a CLA from a given tracker id
is recorded in the tracker.  People also get reminded to submit a CLA
if they haven't yet but have submitted a patch.

> > At best core developers tell a contributor "please send your PR
> > against 3.4", push-button merge it, update a local clone, merge from
> > 3.4 to default, do the usual stuff, commit, and then push;
> 
> Is it common for a patch that should apply to multiple branches to fail
> on some but not all of them?

Currently?  Yes when 2.7 is involved.  If we fix NEWS, then it won't
be *common* for maint->default, but it will happen.

> In other words, is there any reason beyond "not done yet" that submitting
> a patch (or pull request) shouldn't automatically create a patch per
> branch, with pushbuttons to test/reject/commit?

Not Done Yet (by any of the tools we know about) is the only reason I'm
aware of.

> > Our code review tool is a fork that probably should be
> > replaced as only Martin von Loewis can maintain it.
> 
> Only he knows the innards, or only he is authorized, or only he knows
> where the code currently is/how to deploy an update?

Only he knows the innards.  (Although Ezio has made at least one patch
to it.)  I think Guido's point was that we (the community) shouldn't be
maintaining this private fork of a project that has moved on well beyond
us; instead we should be using an active project and leveraging
its community with our own contributions (like we do with Roundup).

> I know that there were times in the (not-so-recent) past when I had
> time and willingness to help with some part of the infrastructure, but
> didn't know where the code was, and didn't feel right making a blind
> offer.

Yeah, that's something that's been getting better lately (thanks,
infrastructure team), but where to get the info is still not as clear as
would be optimal.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition

[R. David Murray]

On Sat, 13 Dec 2014 10:17:59 -0500, Barry Warsaw  wrote:
> On Dec 13, 2014, at 12:29 AM, Donald Stufft wrote:
> 
> >For what it’s worth, I almost exclusively write 2/3 compatible code (and
> >that’s with the “easy” subset of 2.6+ and either 3.2+ or 3.3+) and 
> >doing so
> >does make the language far less fun for me than when I was writing 2.x only
> >code.
> 
> For myself, the way I'd put it is:
> 
> With the libraries I maintain, I generally write Python 2/3 compatible code,
> targeting Python 2.7 and 3.4, with 2.6, 3.3, and 3.2 support as bonuses,
> although I will not contort too much to support those older versions.  Doing
> so does make the language far less fun for me than when I am writing 3.x only
> code.  All applications I write in pure Python 3, targeting Python 3.4, unless
> my dependencies are not all available in Python 3, or I haven't yet had the
> cycles/resources to port to Python 3.  Writing and maintaining applications in
> Python 2 is far less fun than doing so in Python 3.

I think this is an important distinction.  The considerations are very
different for library maintainers than they are for application
maintainers.  Most of my work is in (customer) applications, and except
for one customer who insists on using an old version of RedHat, I've
been on "latest" python3 for those for quite a while now.  I suspect we
hear less here from people in that situation than would be proportional
to their absolute numbers.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition

[R. David Murray]

On Tue, 16 Dec 2014 10:48:07 -0800, Mark Roberts  wrote:
> On Tue, Dec 16, 2014 at 2:45 AM, Antoine Pitrou  wrote:
> >
> > Iterating accross a dictionary doesn't need compatibility shims. It's
> > dead simple in all Python versions:
> >
> > $ python2
> > Python 2.7.8 (default, Oct 20 2014, 15:05:19)
> > [GCC 4.9.1] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> d = {'a': 1}
> > >>> for k in d: print(k)
> > ...
> > a
> >
> > $ python3
> > Python 3.4.2 (default, Oct  8 2014, 13:08:17)
> > [GCC 4.9.1] on linux
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> d = {'a': 1}
> > >>> for k in d: print(k)
> > ...
> > a
> >
> > Besides, using iteritems() and friends is generally a premature
> > optimization, unless you know you'll have very large containers.
> > Creating a list is cheap.
> >
> 
> It seems to me that every time I hear this, the author is basically
> admitting that Python is a toy language not meant for "serious computing"
> (where serious is defined in extremely modest terms). The advice is also
> very contradictory to literally every talk on performant Python that I've
> seen at PyCon or PyData or ... well, anywhere. And really, doesn't it
> strike you as incredibly presumptuous to call the *DEFAULT BEHAVIOR* of
> Python 3 a "premature optimization"?

No.  A premature optimization is one that is made before doing any
performance analysis, so language features are irrelevant to that
labeling.  This doesn't mean you shouldn't use "better" idioms when they
are clear.  But if you are complicating your code because of performance
concerns *without measuring it* you are doing premature optimization, by
definition[*].

> Isn't the whole reason that the
> default behavior switch was made is because creating lists willy nilly all
> over the place really *ISN'T* cheap? This isn't the first time someone has

No.  In Python3 we made the iterator protocol more central to the
language.  Any performance benefit is actually a side effect of that
change.  One that was considered, yes, but in the context of the
*language* as a whole and not any individual program's performance
profile.  And "this doesn't make things worse for real world programs as
far as we can measure" is a more important criterion for this kind of
language change than "lets do this because we've measured and it makes
things better".

--David

[*] And yes, *we all do this*.  Sometimes doing it doesn't cost much.
Sometimes it does.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] contributing to multiprocessing

[R. David Murray]

On Thu, 08 Jan 2015 17:08:07 -0800, Ethan Furman  wrote:
> On 01/08/2015 03:21 PM, Davin Potts wrote:
> > 
> > I am interested in making some serious ongoing contributions around 
> > multiprocessing.
> 
> Great!
> 
> > Rather than me simply walking through that backlog, offering comments or 
> > encouragement here and there on issues, it
> > makes more sense for me to ask:  what is the right way for me to proceed?  
> > What is the next step towards me helping
> > triage issues?  Is there a bridge-keeper with at least three, no more than 
> > five questions for me?
> 
> I would suggest having at least one, if not two or three, current core-devs 
> ready and willing to quickly review your
> work (I believe Raymond Hettinger may be one); then, go ahead and triage, 
> improve and/or submit patches, and make
> comments.  Once you've got a few of these under your belt, with favorable 
> reviews and your patches committed, you may
> get stuck with commit privileges of your own.  ;)

Indeed, the best way to proceed, regardless of any other issues, is in
fact to review, triage, comment on, and improve the issues you are
interested in.  Get the patches commit ready, and then get a current
core dev to do a commit review.

Oddly, the doc issues may be more problematic than the code issues.
Fixing bugs in docs isn't difficult to get done, but restructuring
documentation sometimes gets bogged down in differing opinions.  (I
haven't myself looked at your proposals since I don't use
multiprocessing, so I don't know how radical the proposed changes are).

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[R. David Murray]

On Mon, 12 Jan 2015 17:26:43 +, Steve Dower  
wrote:
> David Anthoff wrote:
> > Yes, those are good examples. Right now doing this in the way these guys do 
> > is
> > too much work for our small project... Anything that makes this easier 
> > would be
> > appreciated.
> 
> I don't see how. All they've done is literally copy a Python
> installation into their install directory. Yes, they have their own
> launcher executables (py2exe generated, it looks like) and have
> precompiled the standard library and put it in a ZIP file, but you
> don't even need to go that far. Without knowing anything about your
> project I can't give specific suggestions, but simply dropping a
> Python installation in is not that difficult (and until the issues
> that Nick referred to are fixed, will have the same problems as
> TortoiseHg presumably has).

That's what py2exe *does*.  It does all the python-integration and
launcher-building work for you.  I use it for a small project myself
(proprietary), which I build an installer for using Inno Setup.  Works
very well, supports python3.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] incorrect docstring for sys.int_info.sizeof_digit?

[R. David Murray]

On Wed, 21 Jan 2015 14:53:19 +, Tim Golden  wrote:
> On 21/01/2015 11:07, Pfeiffer, Phillip E., IV wrote:
> > Apologies if this has already been reported; I couldn't find a
> > readily searchable archive for this mailing list (and apologies if
> > I've just missed it).
> 
> Depending on "readily searchable", this isn't too bad:
> 
> http://markmail.org/search/?q=list%3Aorg.python.python-dev+integer+docstring

But, if you are searching to see if a bug has been reported, you want to
search the tracker at bugs.python.org, not the python-dev mailing list.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] rst files

[R. David Murray]

On Fri, 23 Jan 2015 15:55:29 -0800, Guido van Rossum  wrote:
> This adds entries to the index of the document -- similar to the index at
> the end of a book. I think single vs. double refers to different types of
> entries. Check out this page: https://docs.python.org/3/genindex.html
> 
> On Fri, Jan 23, 2015 at 3:43 PM, Ethan Furman  wrote:
> 
> > Can somebody please explain this?
> >
> > .. index::
> >single: formatting, string (%)
> >single: interpolation, string (%)
> >single: string; formatting
> >single: string; interpolation
> >single: printf-style formatting
> >single: sprintf-style formatting
> >single: % formatting
> >single: % interpolation
> >
> > Specifically, what does index mean?  What does single vs double vs triple
> > mean?  Is there a reference somewhere I can read?

It is explained in the Sphinx documentation:
http://sphinx-doc.org/contents.html

Specifically:

http://sphinx-doc.org/markup/misc.html#directive-index

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why are generated files in the repository?

[R. David Murray]

On Sun, 25 Jan 2015 14:00:57 +1000, Nick Coghlan  wrote:
> It's far more developer friendly to aim to have builds from a source
> check-out "just work" if we can. That's pretty much where we are today
> (getting external dependencies for the optional parts on *nix can still be
> a bit fiddly - it may be worth maintaining instructions for at least apt
> and yum in the developer guide that cover that)

https://docs.python.org/devguide/setup.html#build-dependencies
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any grammar experts?

[R. David Murray]

On Mon, 26 Jan 2015 01:21:24 +0100, Antoine Pitrou  wrote:
> On Sun, 25 Jan 2015 14:59:42 -0800
> Guido van Rossum  wrote:
> > On Sun, Jan 25, 2015 at 7:32 AM, Georg Brandl  wrote:
> > 
> > > On 01/25/2015 04:08 PM, Antoine Pitrou wrote:
> > > > On Sat, 24 Jan 2015 21:10:51 -0500
> > > > Neil Girdhar  wrote:
> > > >> To finish PEP 448, I need to update the grammar for syntax such as
> > > >>
> > > >> {**x for x in it}
> > > >
> > > > Is this seriously allowed by the PEP? What does it mean exactly?
> > >
> > > It appears to go a bit far.  Especially since you also would have to allow
> > >
> > > {*x for x in it}
> > >
> > > which is a set comprehension, while the other is a dict comprehension :)
> > >
> > 
> > That distinction doesn't bother me -- you might as well claim it's
> > confusing that f(*x) passes positional args from x while f(**x) passes
> > keyword args.
> > 
> > And the varargs set comprehension is similar to the varargs list
> > comprehension:
> > 
> > [*x for x in it]
> > 
> > If `it` were a list of three items, this would be the same as
> > 
> > [*it[0], *it[1], *it[2]]
> 
> I find all this unreadable and difficult to understand.

I did too, before reading the PEP.

After reading the PEP, it makes perfect sense to me.  Nor is the PEP
complicated...it's just a matter of wrapping your head around the
generalization[*] of what are currently special cases that is going on
here.

--David

[*] The *further* generalization...we've already had, for example,
the generalization that allows:

a, *b = (1, 3, 4)

to work, and that seems very clear to usnow.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any grammar experts?

[R. David Murray]

On Mon, 26 Jan 2015 09:43:26 -0500, Barry Warsaw  wrote:
> On Jan 25, 2015, at 09:31 PM, R. David Murray wrote:
> 
> >> > > {*x for x in it}
> >> > >
> >> > > which is a set comprehension, while the other is a dict comprehension 
> >> > > :)
> >> > >
> >> > 
> >> > That distinction doesn't bother me -- you might as well claim it's
> >> > confusing that f(*x) passes positional args from x while f(**x) passes
> >> > keyword args.
> >> > 
> >> > And the varargs set comprehension is similar to the varargs list
> >> > comprehension:
> >> > 
> >> > [*x for x in it]
> >> > 
> >> > If `it` were a list of three items, this would be the same as
> >> > 
> >> > [*it[0], *it[1], *it[2]]
> >> 
> >> I find all this unreadable and difficult to understand.
> >
> >I did too, before reading the PEP.
> >
> >After reading the PEP, it makes perfect sense to me.  Nor is the PEP
> >complicated...it's just a matter of wrapping your head around the
> >generalization[*] of what are currently special cases that is going on
> >here.
> 
> It does make sense after reading the PEP but it also reduces the readability
> and instant understanding of any such code.  This is head-scratcher code that
> I'm sure I'd get asked about from folks who aren't steeped in all the dark
> corners of Python.  I don't know if that's an argument not to adopt the PEP,
> but it I think it would be a good reason to recommend against using such
> obscure syntax, unless there's a good reason (and good comments!).

But it is only obscure because we are not used to it yet.  It is a
logical extension of Python's existing conventions once you think about
it.  It will become "obvious" quickly, IMO.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any grammar experts?

[R. David Murray]

On Mon, 26 Jan 2015 22:05:44 +0100, Antoine Pitrou  wrote:
> On Mon, 26 Jan 2015 12:22:20 -0800
> Ethan Furman  wrote:
> > On 01/26/2015 12:09 PM, Antoine Pitrou wrote:
> > > On Mon, 26 Jan 2015 12:06:26 -0800
> > > Ethan Furman  wrote:
> > >> It destroy's the chaining value and pretty much makes the improvement 
> > >> not an improvement.  If there's a possibility that
> > >> the same key could be in more than one of the dictionaries then you 
> > >> still have to do the
> > >>
> > >>   dict.update(another_dict)
> > > 
> > > So what? Is the situation where chaining is desirable common enough?
> > 
> > Common enough to not break it, yes.
> 
> Really? What are the use cases?

My use case is a configuration method that takes keyword parameters.
In tests I want to specify a bunch of default values for the
configuration, but I want individual test methods to be able
to override those values.  So I have a bunch of code that does
the equivalent of:

from test.support import default_config
[...]
def _prep(self, config_overrides):
config = default.config.copy()
config.update(config_overrides)
my_config_object.load(**config)


With the current proposal I could instead do:

def _prep(self, config_overrides):
my_config_object.load(**default_config, **config_overrides)

I suspect I have run into situations like this elsewhere as well, but
this is the one from one of my current projects.

That said, I must admit to being a bit ambivalent about this, since
we otherwise are careful to disallow duplicate argument names.

So, instead we could write:

my_config_object.load(**{**default_config, **config_overrides})

since dict literals *do* allow duplicate keys.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any grammar experts?

[R. David Murray]

On Tue, 27 Jan 2015 00:07:08 +0100, Antoine Pitrou  wrote:
> On Mon, 26 Jan 2015 16:28:24 -0500
> "R. David Murray"  wrote:
> > 
> > My use case is a configuration method that takes keyword parameters.
> > In tests I want to specify a bunch of default values for the
> > configuration, but I want individual test methods to be able
> > to override those values.  So I have a bunch of code that does
> > the equivalent of:
> > 
> > from test.support import default_config
> > [...]
> > def _prep(self, config_overrides):
> > config = default.config.copy()
> > config.update(config_overrides)
> > my_config_object.load(**config)
> > 
> > 
> > With the current proposal I could instead do:
> > 
> > def _prep(self, config_overrides):
> > my_config_object.load(**default_config, **config_overrides)
> 
> It sounds like the _prep() method exists once in your code base, this
> isn't an idiom you are duplicating everywhere. The incentive for a
> syntactic shortcut looks pretty thin.

Something similar exists between five and ten times (I didn't go in and
count) in my current code base, in various specific forms for different
test classes.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] TypeError messages

[R. David Murray]

On Sun, 22 Feb 2015 00:26:23 +1000, Nick Coghlan  wrote:
> On 21 February 2015 at 00:05, Brett Cannon  wrote:
> >  I agree that type names don't need to be quoted.
> 
> As a user, I actually appreciate the quotes, because they make the
> error pattern easier for me to parse. Compare:
> 
> int expected, but found str
> float expected, but found int
> 
> 'int' expected, but found 'str'
> 'float' expected, but found 'int'

It's not a big deal to me either way, but I find the version with the
quotes makes me think it is looking for the literal string 'int', but
found the literal string 'str', whereas in the first case it seems
clear it is looking for objects of that type.  Perhaps this is just
because I am used to the existing messages, but I think it goes
beyond that.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Not being able to compile: "make: *** [Programs/_freeze_importlib] Error 1"

[R. David Murray]

On Thu, 16 Apr 2015 18:09:01 -0300, Facundo Batista  
wrote:
> Full trace here:
> 
>   http://linkode.org/TgkzZw90JUaoodvYzU7zX6
> 
> Before going into a deep debug, I thought about sending a mail to see
> if anybode else hit this issue, if it's a common problem, if there's a
> known workaround.

Most likely you just need to run 'make touch' so that it doesn't try
to rebuild stuff it doesn't need to (because we check in those
particular build artifacts, like the frozen importlib).

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Summary of Python tracker Issues

[R. David Murray]

On Fri, 17 Apr 2015 18:08:24 +0200, Python tracker  
wrote:
> 
> ACTIVITY SUMMARY (2015-04-10 - 2015-04-17)
> Python tracker at http://bugs.python.org/
> 
> To view or respond to any of the issues listed below, click on the issue.
> Do NOT respond to this message.
> 
> Issues counts and deltas:
>   open4792 (-31)
>   closed 30957 (+113)
>   total  35749 (+82)

That's a successful sprint week :)

Thanks everyone!

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

I wrote a longer response and then realized it didn't really add much to
the discussion.  So let me be short: type annotations do *not* appeal to
me, and I am not looking forward to the cognitive overhead of dealing
with them.  Perhaps I will eventually grow to like them if the tools
that use them really add value.  You'll have to sell me on it, though.

On Mon, 20 Apr 2015 12:35:33 -0700,  wrote:
> Stub files have many downsides, too, unfortunately:
> - we don’t *want* to have them, but we *need* to have them (C extensions, 
> third-party modules, Python 2, …)
> - they bring cognitive overhead of having to switch between two files
> - they require the author to repeat himself quite a lot
> - they might go out of date much easier than annotations in the function 
> signature

The whole point of type hints is for the linters/IDEs, so IMO it is
perfectly reasonable to put the burden of making them useful onto the
linters/IDEs.  The UI for it can unify the two files into a single
view...I know because way back in the dark ages I wrote a small
editor-based IDE that did something very analogous on an IBM Mainframe,
and it worked really well as a development environment.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

+1 to this from me too. I'm afraid that means I'm -1 on the PEP.

I didn't write this in my earlier email because I wasn't sure about it,
but my gut reaction after reading Harry's email was "if type annotations
are used in the stdlib, I'll probably stop contributing".  That doesn't
mean that's *true*, but that's the first time I've ever had that
thought, so it is probably worth sharing.

Basically, it makes Python less fun to program in.  That may be be an
emotional reaction and irrational, but I think it matters.  And yes, I
write production Python code for a living, though granted not at Google
or Facebook or Dropbox scale.

On Mon, 20 Apr 2015 19:00:53 -0500, Ryan Gonzalez  wrote:
> Although I like the concept of type annotations and the PEP, I have to
> agree with this. If I saw these type annotations when learning Python (I'm
> self-taught), there's a 99% chance I would've freaked.
> 
> It's the same issue as with teaching C++: it's wrong to say, "Hey, I taught
> you the basics, but there's other stuff that's going to confuse you to a
> ridiculous extent when you read it." People can't ignore it. It'll become a
> normal part of Python programs.
> 
> At least now you can say, "I'm using the mypy type checker."
> 
> Don't get me wrong; I like mypy. I helped with their documentation and am
> watching the GitHub repo. But this is dead-on.
> 
> 
> On Mon, Apr 20, 2015 at 6:41 PM, Jack Diederich  wrote:
> 
> > Twelve years ago a wise man said to me "I suggest that you also propose a
> > new name for the resulting language"
> >
> > I talked with many of you at PyCon about the costs of PEP 484. There are
> > plenty of people who have done a fine job promoting the benefits.
> >
> > * It is not optional. Please stop saying that. The people promoting it
> > would prefer that everyone use it. If it is approved it will be optional in
> > the way that PEP8 is optional. If I'm reading your annotated code it is
> > certainly /not/ optional that I understand the annotations.
> >
> > * Uploading stubs for other people's code is a terrible idea. Who do I
> > contact when I update the interface to my library? The random Joe who
> > "helped" by uploading annotations three months ago and then quit the
> > internet? I don't even want to think about people maliciously adding stubs
> > to PyPI.
> >
> > * The cognitive load is very high. The average function signature will
> > double in length. This is not a small cost and telling me it is "optional"
> > to pretend that every other word on the line doesn't exist is a farce.
> >
> > * Every company's style guide is about to get much longer. That in itself
> > is an indicator that this is a MAJOR language change and not just some
> > "optional" add-on.
> >
> > * People will screw it up. The same people who can't be trusted to program
> > without type annotations are also going to be *writing* those type
> > annotations.
> >
> > * Teaching python is about to get much less attractive. It will not be
> > optional for teachers to say "just pretend all this stuff over here doesn't
> > exist"
> >
> > * "No new syntax" is a lie. Or rather a red herring. There are lots of new
> > things it will be required to know and just because the compiler doesn't
> > have to change doesn't mean the language isn't undergoing a major change.
> >
> > If this wasn't in a PEP and it wasn't going to ship in the stdlib very few
> > people would use it. If you told everyone they had to install a different
> > python implementation they wouldn't. This is much worse than that - it is
> > Python4 hidden away inside a PEP.
> >
> > There are many fine languages that have sophisticated type systems. And
> > many bondage & discipline languages that make you type things three times
> > to make really really sure you meant to type that. If you find those other
> > languages appealing I invite you to go use them instead.
> >
> > -Jack
> >
> > https://mail.python.org/pipermail/python-dev/2003-February/033291.html
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
> >
> >
> 
> 
> -- 
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something’s wrong.
> http://kirbyfan64.github.io/
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

On Wed, 22 Apr 2015 01:09:52 +1000, Chris Angelico  wrote:
> def incremental_parser(input: FileLike) -> List[Token]:
> tokens = []
> data = ""
> while True:
> if not data:
> data = input.read(64)
> token = Token(data[0]); data = data[1:]
> while token.wants_more():
> token.give_more(data[0]); data = data[1:]
> tokens.append(token)
> if token.is_end_of_stream(): break
> input.seek(-len(data), 1)
> return tokens
> 
> If you were to exhaustively stipulate the requirements on the
> file-like object, you'd have to say:
> 
> * Provides a read() method which takes an integer and returns up to
> that many bytes
> * Provides a seek() method which takes two integers
> * Is capable of relative seeking by at least 63 bytes backward
> * Is open for reading
> * Etcetera
> 
> That's not the type system's job. Not in Python. Maybe in Haskell, but

Just a note that if I'm reading the high level description right,
this kind of analysis is exactly the kind of thing that Flow does
for javascript.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

On Tue, 21 Apr 2015 18:27:50 +0300, Paul Sokolovsky  wrote:
> > I was replying to Steven's message. Did you read it?
> 
> Yes. And I try to follow general course of discussion, as its hard to
> follow individual sub-threads. And for example yesterday's big theme
> was people blackmailing that they stop contributing to stdlib if
> annotations are in, and today's theme appear to be people telling that
> static type checking won't be useful. And just your reply to Steven
> was a final straw which prompted me to point out that static type
> checking is not a crux of it, but just one (not the biggest IMHO) usage.

Please be respectful rather than inflammatory.  If you read what I
wrote, I did not say that I was going to stop contributing, I
specifically talked about that gut reaction being both emotional and
illogical.  That doesn't make the reaction any less real, and the fact
that such reactions exist is a data point you should consider in
conducting your PR campaign for this issue.  (I don't mean that last as
a negative:  this issue *requires* an honest PR campaign.)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

On Tue, 21 Apr 2015 16:55:49 -, "Gregory P. Smith"  wrote:
> We will not be putting type annotations anywhere in the stdlib or expecting
> anyone else to maintain them there. That would never happen until tools
> that are convincing enough in their utility for developers to _want_ to use
> are available and accepted.  That'll be a developer workflow thing we could
> address with a later PEP. IF it happens at all.

This is the most reassuring statement I've heard so far ;)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

On Tue, 21 Apr 2015 10:10:06 -0700, Guido van Rossum  wrote:
> On Tue, Apr 21, 2015 at 9:17 AM, R. David Murray 
> wrote:
> 
> > Please be respectful rather than inflammatory.  If you read what I
> > wrote, I did not say that I was going to stop contributing, I
> > specifically talked about that gut reaction being both emotional and
> > illogical.  That doesn't make the reaction any less real, and the fact
> > that such reactions exist is a data point you should consider in
> > conducting your PR campaign for this issue.  (I don't mean that last as
> > a negative:  this issue *requires* an honest PR campaign.)
> >
> 
> Well, my own reactions at this point in the flame war are also quite
> emotional. :-(
> 
> I have done my best in being honest in my PR campaign. But I feel like the

I believe you have.  My inclusion of the word 'honest' was meant to
contrast the kind of PR we need with the kind of PR people typically
think about, which is often not particularly honest.

> opposition (not you, but definitely some others -- have you seen Twitter?)
> are spreading FUD based on an irrational conviction that this will destroy

No, I tend only to peek in there occasionally.

> Python. It will not. It may not prove the solution to all Python's problems
> -- there's always 3.6. (Oh wait, Python 2.7 is perfect. I've heard that
> before -- Paul Everitt famously said the same of Python 1.5.2. Aren't you
> glad I didn't take him literally? :-P )

Yes.  But somewhere there or not long before was my introduction to
Python, so I remember it fondly :)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[R. David Murray]

On Tue, 21 Apr 2015 21:31:49 +0300, Paul Sokolovsky  wrote:
> On Tue, 21 Apr 2015 09:50:59 -0700 Ethan Furman  wrote:
> 
> > On 04/21, Paul Sokolovsky wrote:
> > > 
> > > And for example yesterday's big theme was people blackmailing that
> > > they stop contributing to stdlib if annotations are in [...]
> > 
> > A volunteer's honest reaction is not blackmail, and categorizing it
> > as such is not helpful to the discussion.
> 
> Sure, that was rather humoresque note. Still, one may wonder why
> "honest reaction" is like that, if from reading PEP484 it's clear that
> it doesn't change status quo: https://www.python.org/dev/peps/pep-3107
> added annotations long ago, and PEP484 just provides default

But what concerned me as a Python core developer was the perception that
as a core developer I would have to deal with type hints and type
checking in the stdlib, because there was talk of including type hints
for the stdlib (as stub files) in 3.6.  *That* was the source of my
concern, which is reduced by the statement that we'll only need to think
about type checking in the stdlib once the tools are *clearly* adding
value, in *our* (the stdlib module maintainers') opinion, so that
we *want* to use the type hints.

The discussion of stub files being maintained by interested parties in a
separate repository is also helpful.  Again, that means that wider
adoption should mostly only happen if the developers see real benefit.
I still dislike the idea of having to read type hints (I agree with
whoever it was said a well written docstring is more helpful than
abstract type hints when reading Python code), but perhaps I will get
used to it if/when it shows up in libraries I use.

I think it would serve your goal better if instead of dismissing
concerns you were to strive to understand them and then figure out how
to allay them.

Overall, I suspect that those who are doubtful are put off by the "rah
rah this is great you are fools for not wanting to use it" tone of
(some) of the proponents (which makes us think this is going to get
pushed on us willy-nilly), whereas hearing a consistent message that
"this will *enable* interested parties to collaborate on type checking
without impacting the way the rest of you code *unless it proves its
value to you*" would be better received.  Guido has done the latter,
IMO, as have some others.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

[Sven R. Kunze]

Why not adding both? Properties do have their uses as does __getattr__.

Cheers,
Sven

On 13.09.2017 11:43, Larry Hastings wrote:


On 09/12/2017 12:38 AM, Larry Hastings wrote:

On 09/11/2017 07:22 PM, Guido van Rossum wrote:


The prototype is linked to from the PEP; for your convenience
here's a link:

https://github.com/larryhastings/cpython/tree/module-properties


I found that link in the PEP, but it's just a branch of a fork of 
cpython. It would be easier to review the prototype as a PR to 
upstream cpython.


Okay, I'll make a PR tomorrow and post a reply here (and update the PEP).


PR is here:

https://github.com/python/cpython/pull/3534


Cheers,


//arry/


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Sven R. Kunze]

Thanks for the PEP! :)

I like the naming. ;) Though, I would like to add to Michel's argument 
in favor of a base class.



On 11.09.2017 08:38, Michel Desmoulin wrote:

- I read Guido talking about some base class as alternative to the
generator version, but don't see it in the PEP. Is it still considered ?

I'm going to put some words in explaining why I don't want to use base
classes (I don't think it buys you anything). Do you have a reason for
preferring base classes?

Not preferring, but having it as an alternative. Mainly for 2 reasons:

1 - data classes allow one to type in classes very quickly, let's
harvest the benefit from that.

Typing a decorator in a shell is much less comfortable than using
inheritance. Same thing about IDE: all current ones have snippet with
auto-switch to the class parents on tab.

All in all, if you are doing exploratory programming, and thus
disposable code, which data classes are fantastic for, inheritance will
keep you in the flow.

2 - it will help sell the data classes

I train a lot of people to Python each year. I never have to explain
classes to people with any kind of programming background. I _always_
have to explain decorators.

People are not used to it, and even kind fear it for quite some time.

Inheritance however, is familiar, and will not only push people to use
data classes more, but also will let them do less mistakes: they know
the danger of parent ordering, but not the ones of decorators ordering.


3)  - the order of base classes can arranged appropriately

In our day-to-day work, we use mixins and cooperative multiple 
inheritance a lot.

So, having dataclasses as a base class or a mixin would be great! :)

Combined with 1) and 2), I am much in favor of having dataclasses as 
base class/mixin than as a decorator.

What are the benefits of the decorator? Maybe both is possible?

Cheers,
Sven


PS: @Michel good observation 1). Typing decorators in shell is annoying.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Sven R. Kunze]

On 03.11.2017 21:34, Paul Moore wrote:

Consider someone who's downloaded Python and PyCharm (or Visual
Studio). They want to get the benefit of the IDE code completion
facilities, so they declare their argument as List[int], following the
information they've found on how to declare lists of integers.


The PyCharm I know is capable of detecting such simple types on its own, 
without type hints.



I for one like the idea of a faster evolution of typing.py.

Cheers,
Sven


PS: pip is pretty standard these days, so I don't think it's much of an 
issue for guys who really needs it installed.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Sven R. Kunze]

+1 from me too.


On 04.11.2017 21:55, Jim Baker wrote:
+1, as Guido correctly recalls, this language guarantee will work well 
with Jython when we get to the point of implementing 3.7+.


On Sat, Nov 4, 2017 at 12:35 PM, Guido van Rossum > wrote:


This sounds reasonable -- I think when we introduced this in 3.6
we were worried that other implementations (e.g. Jython) would
have a problem with this, but AFAIK they've reported back that
they can do this just fine. So let's just document this as a
language guarantee.

On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah mailto:ste...@bytereef.org>> wrote:


Hello,

would it be possible to guarantee that dict literals are
ordered in v3.7?


The issue is well-known and the workarounds are tedious, example:

https://mail.python.org/pipermail/python-ideas/2015-December/037423.html




If the feature is guaranteed now, people can rely on it around
v3.9.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org





-- 
--Guido van Rossum (python.org/~guido )


___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:
https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allow annotations using basic types in the stdlib?

[R. David Murray]

I agree with Steve.  There is *cognitive* overhead to type annotations.
I find that they make Python code harder to read and understand.  So I
object to them in the documentation and docstrings as well.  (Note:
while I agree that the notation is compact for the simple types, the
fact that it would appear for some signatures and not for others is a
show stopper from my point of view...consistency is important to reducing
the cognitive overhead of reading the docs.)

I'm dealing with the spread of annotations on my current project,
having to ask programmers on the team to delete annotations that they've
"helpfully" added that to my mind serve no purpose on a project of the
size we're developing, where we aren't using static analysis for anything.

Maybe I'm being a curmudgeon standing in the way of progress, but I'm
pretty sure there are a number of people in my camp :)

On Mon, 06 Nov 2017 16:22:23 +, Steve Holden  wrote:
> While I appreciate the value of annotations I think that *any* addition of
> them to the stdlib would complicate an important learning resource
> unnecessarily. S
> 
> Steve Holden
> 
> On Mon, Nov 6, 2017 at 4:07 PM, Victor Stinner 
> wrote:
> 
> > Related to annotations, are you ok to annotate basic types in the
> > *documentation* and/or *docstrings* of the standard library?
> >
> > For example, I chose to document the return type of time.time() (int)
> > and time.time_ns() (float). It's short and I like how it's formatted.
> > See the current rendered documentation:
> >
> > https://docs.python.org/dev/library/time.html#time.time
> >
> > "Annotations" in the documentation and docstrings have no impact on
> > Python runtime performance. Annotations in docstrings makes them a few
> > characters longer and so impact the memory footprint, but I consider
> > that the overhead is negligible, especially when using python3 -OO.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Sven R. Kunze]

Isn't yield like a return?

A return in a list/dict/set comprehension makes no sense to me.

So, +1 on SyntaxError from me too.

Cheers.


On 22.11.2017 21:29, David Mertz wrote:
Inasmuch as I get to opine, I'm +1 on SyntaxError. There is no 
behavior for that spelling that I would find intuitive or easy to 
explain to students. And as far as I can tell, the ONLY time anything 
has ever been spelled that way is in comments saying "look at this 
weird edge case behavior in Python."


On Nov 22, 2017 10:57 AM, "Jelle Zijlstra" > wrote:




2017-11-22 9:58 GMT-08:00 Guido van Rossum mailto:gu...@python.org>>:

Wow, 44 messages in 4 hours. That must be some kind of record.

If/when there's an action item, can someone summarize for me?

The main disagreement seems to be about what this code should do:

    g = [(yield i) for i in range(3)]

Currently, this makes `g` into a generator, not a list. Everybody
seems to agree this is nonintuitive and should be changed.

One proposal is to make it so `g` gets assigned a list, and the
`yield` happens in the enclosing scope (so the enclosing function
would have to be a generator). This was the way things worked in
Python 2, I believe.

Another proposal is to make this code a syntax error, because it's
confusing either way. (For what it's worth, that would be my
preference.)

There is related discussion about the semantics of list
comprehensions versus calling list() on a generator expression,
and of async semantics, but I don't think there's any clear point
of action there.

-- 
--Guido van Rossum (python.org/~guido

)

___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:

https://mail.python.org/mailman/options/python-dev/jelle.zijlstra%40gmail.com





___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:
https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Sven R. Kunze]

On 23.11.2017 08:38, Ivan Levkivskyi wrote:

I think this code should be just equivalent to this code

    def g():
    temp = [(yield i) for i in range(10)]
    return (v for v in temp)

Semantics of the comprehension should be clear here (just an 
equivalent for-loop without name leaking)


Excuse me if I disagree here. If I were to understand this in real-world 
code, I cannot imagine what will happen here.


A "yield" within a comprehension is like a "return" in a comprehension. 
It makes no sense at all.

Also a "yield" and a "return with value" is also rarely seen.

Comprehensions build new objects, they are not for control flow, IMO.

Cheers,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Sven R. Kunze]

On 21.12.2017 11:22, Terry Reedy wrote:



@dataclass
class C:
 a: int # integer field with no default
 b: float = 0.0 # float field with a default

And the types will be recognized by type checkers such as mypy.

And I think the non-typed examples should go first in the docs.




I still don't understand why "I don't care" can be defined by "leaving out"

@dataclass
class C:
 b = 0.0 # float field with a default


For non-default fields, I like ellipsis too.

Cheer,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com