Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Sun, Jun 22, 2014 at 8:00 AM, Steve Dower wrote: > We can always lie about the version in sys.version. Existing code is > unaffected and new code will have to use version_info (Windows developers > will know that Windows pulls tricks like this every other version... doesn't > make it a great idea, but it works). I'd prefer a change of format to an outright lie. Something like "2.7._10" which will sort after "2.7.9". But ideally, nothing at all - just go smoothly to "2.7.10" and let broken code be broken. It'll think it's running on 2.7.1, and if anything needs to distinguish between 2.7.1 and 2.7.x, hopefully it's using version_info. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Sun, Jun 22, 2014 at 7:37 AM, M.-A. Lemburg wrote: > There are no places in the stdlib that parse sys.version in a > way that would break wtih 2.7.10, AFAIK. I was just referring > to the statement that Nick quoted. sys.version *is* used for > parsing the Python version or using parts of it to build > e.g. filenames and that's really no surprise. Right, good to know. I thought you were implying that stuff would break. Yes, stuff definitely does parse out the version number from sys.version, lots of that happens. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 patch levels turning two digit
On Tue, Jun 24, 2014 at 6:42 AM, "Martin v. Löwis" wrote: > See my other message. It's actually heavier, since it requires changes > to distutils, PyPI, pip, buildout etc., all which know how to deal with > Python minor version numbers, but are unaware of the notion of competing > ABIs on Windows (except that they know how to deal with 32-bit vs. 64-bit). Is it possible to hijack the "deal with 32-bit vs 64-bit"ness of things to handle the different compilers? So, for instance, there might be a "32-bit-NewCompiler" and a "64-bit-NewCompiler", two new architectures, just as if someone came out with a 128-bit Windows and built Python 2.7 for it. Would packaging be able to handle that more easily than a compiler change within the same architecture? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fix Unicode-disabled build of Python 2.7
On Thu, Jun 26, 2014 at 9:04 PM, Antoine Pitrou wrote: > For the same reason, I agree with Victor that we should ditch the > threading-disabled builds. It's too much of a hassle for no actual, > practical benefit. People who want a threadless unicodeless Python can > install Python 1.5.2 for all I care. Or some other implementation of Python. It's looking like micropython will be permanently supporting a non-Unicode build (although I stepped away from the project after a strong disagreement over what would and would not make sense, and haven't been following it since). If someone wants a Python that doesn't have stuff that the core CPython devs treat as essential, s/he probably wants something like uPy anyway. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator
On Sat, Jun 28, 2014 at 11:05 PM, Akira Li <4kir4...@gmail.com> wrote: > Have you considered adding support for paths relative to directory > descriptors [1] via keyword only dir_fd=None parameter if it may lead to > more efficient implementations on some platforms? > > [1]: https://docs.python.org/3.4/library/os.html#dir-fd Potentially more efficient and also potentially safer (see 'man openat')... but an enhancement that can wait, if necessary. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My summary of the scandir (PEP 471)
On Wed, Jul 2, 2014 at 7:20 AM, Paul Moore wrote: > I think that thin wrapper is needed - even > if the various bells and whistles are useful, they can be built on top > of a low-level version (whereas the converse is not the case). +1. Make everything as simple as possible (but no simpler). ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Tue, Jul 8, 2014 at 1:15 AM, Benjamin Peterson wrote: > Why do you think that? > > % python > Python 2.7.6 (default, May 29 2014, 22:22:15) > [GCC 4.7.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. class x(object): pass > ... class y(object): pass > ... x != y > True x == y > False Your analysis is flawed - you're testing the equality of the types, not of instances. But your conclusion's correct; testing instances does work the same way you're implying: rosuav@sikorsky:~$ python Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class x(object): pass ... >>> class y(object): pass ... >>> x() != y() True >>> x() == y() False >>> x() == x() False >>> z = x() >>> z == z True ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Tue, Jul 8, 2014 at 11:59 AM, Rob Cliffe wrote: > If I came across an int object and had no concept of what an integer number > was, how would I know what its "value" is supposed to be? The value of an integer is the number it represents. In CPython, it's entirely possible to have multiple integer objects (ie objects with unique identities) with the same value, although AIUI there are Pythons for which that's not the case. The value of a float, Fraction, Decimal, or complex is also the number it represents, so when you compare 1==1.0, the answer is that they have the same value. They can't possibly have the same identity (every object has a single type), but they have the same value. But what *is* that value? It's not something that can be independently recognized, because casting to a different type might change the value: >>> i = 2**53+1 >>> f = float(i) >>> i == f False >>> f == int(f) True Ergo the comparison of a float to an int cannot be done by casting the int to float, nor by casting the float to int; it has to be done by comparing the abstract numbers represented. Those are the objects' values. But what's the value of a sentinel object? _SENTINEL = object() def f(x, y=_SENTINEL): do_something_with(x) if y is not _SENTINEL: do_something_with(y) I'd say this is a reasonable argument for the default object value to be identity. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Tue, Jul 8, 2014 at 1:12 PM, Steven D'Aprano wrote: > Why? What value (pun intended) is there in adding an explicit statement > of value to every single class? > > "The value of a str is the str's sequence of characters." > "The value of a list is the list's sequence of items." > "The value of an int is the int's numeric value." > "The value of a float is the float's numeric value, or in the case of > INFs and NANs, that they are an INF or NAN." > "The value of a complex number is the ordered pair of its real and > imaginary components." > "The value of a re MatchObject is the MatchObject itself." > > I don't see any benefit to forcing all classes to explicitly document > this sort of thing. It's nearly always redundant and unnecessary. It's important where it's not obvious. For instance, two lists with the same items are equal, two tuples with the same items are equal, but a list and a tuple with the same items aren't. Doesn't mean it necessarily has to be documented, though. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Tue, Jul 8, 2014 at 5:01 PM, Stephen J. Turnbull wrote:
> I agree with Steven d'A that this rule is not part of the language
> definition and shouldn't be, but it's the rule of thumb I find hardest
> to imagine *ever* wanting to break in my own code (although I sort of
> understand why the IEEE 754 committee found they had to).
The reason NaN isn't equal to itself is because there are X bit
patterns representing NaN, but an infinite number of possible
non-numbers that could result from a calculation. Is
float("inf")-float("inf") equal to float("inf")/float("inf")? There
are three ways NaN equality could have been defined:
1) All NaNs are equal, as if NaN is some kind of "special number".
2) NaNs are equal if they have the exact same bit pattern, and unequal else.
3) All NaNs are unequal, even if they have the same bit pattern.
The first option is very dangerous, because it'll mean that "NaN
pollution" can actually result in unexpected equality. The second
looks fine - a NaN is equal to itself, for instance - but it suffers
from the pigeonhole problem, in that eventually you'll have two
numbers which resulted from different calculations and happen to have
the same bit pattern. The third is what IEEE went with. It's the
sanest option.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Tue, Jul 8, 2014 at 5:53 PM, Stephen J. Turnbull wrote: > But you're missing at least two alternatives that > involve raising on some calculations involving NaN, as well as the > fact that forcing inequality of two NaNs produced by equivalent > calculations is arguably just as wrong as allowing equality of two > NaNs produced by the different calculations. This is off-topic for this thread, but still... The trouble is that your "arguably just as wrong" is an indistinguishable case. If you don't want two different calculations' NaNs to *ever* compare equal, the only solution is to have all NaNs compare unequal - otherwise, two calculations might happen to produce the same bitpattern, as there are only a finite number of them available. > That's where things get > fuzzy for me -- in Python I would expect that preserving invariants > would be more important than computational efficiency, but evidently > it's not. What invariant is being violated for efficiency? As I see it, it's one possible invariant (things should be equal to themselves) coming up against another possible invariant (one way of generating NaN is unequal to any other way of generating NaN). Raising an exception is, of course, the purpose of signalling NaNs rather than quiet NaNs, which is a separate consideration from how they compare. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x
On Wed, Jul 9, 2014 at 3:00 AM, Steven D'Aprano wrote: > On Tue, Jul 08, 2014 at 04:58:33PM +0200, Anders J. Munch wrote: > >> For two NaNs computed differently to compare equal is no worse than 2+2 >> comparing equal to 1+3. You're comparing values, not their history. > > a = -23 > b = -42 > if log(a) == log(b): > print "a == b" That could also happen from rounding error, though. >>> a = 2.0**52 >>> b = a+1.0 >>> a == b False >>> log(a) == log(b) True Any time you do any operation on numbers that are close together but not equal, you run the risk of getting results that, in finite-precision floating point, are deemed equal, even though mathematically they shouldn't be (two unequal numbers MUST have unequal logarithms). ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Another case for frozendict
On Mon, Jul 14, 2014 at 12:04 AM, Jason R. Coombs wrote:
> I can achieve what I need by constructing a set on the ‘items’ of the dict.
>
set(tuple(doc.items()) for doc in res)
>
> {(('n', 1), ('err', None), ('ok', 1.0))}
This is flawed; the tuple-of-tuples depends on iteration order, which
may vary. It should be a frozenset of those tuples, not a tuple. Which
strengthens your case; it's that easy to get it wrong in the absence
of an actual frozendict.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x - list delegation to members?
On Mon, Jul 14, 2014 at 2:23 AM, Steven D'Aprano wrote:
>> We will see
>> later that that happens. Further, when comparing float NaNs of the same
>> identity, the list implementation forgot to special-case NaNs. Which
>> would be a bug, IMHO.
>
> "Forgot"? I don't think the behaviour of list comparisons is an
> accident.
Well, "forgot" is on the basis that the identity check is intended to
be a mere optimization. If that were the case ("don't actually call
__eq__ when you reckon it'll return True"), then yes, failing to
special-case NaN would be a bug. But since it's intended behaviour, as
explained further down, it's not a bug and not the result of
forgetfulness.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] == on object tests identity in 3.x - list delegation to members?
On Mon, Jul 14, 2014 at 4:11 AM, Nick Coghlan wrote: > What we've never figured out is a good place to *document* it. I > thought there was an open bug for that, but I can't find it right now. Yeah. The Py3 docs explain why "x in [x]" is True, but I haven't found a parallel explanation of sequence equality. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] class Foo(object) vs class Foo: should be clearly explained in python 2 and 3 doc
On Sun, Aug 10, 2014 at 10:44 AM, Steven D'Aprano wrote: > Looking at your comment here: > >> [1]: https://news.ycombinator.com/item?id=8154471 > > there is a reply from zeckalpha, who says: > >"Actually, leaving out `object` is the preferred convention for > Python 3, as they are semantically equivalent." > > How does (s)he justify this claim? > >"Explicit is better than implicit." > > which is not logical. If you leave out `object`, that's implicit, not > explicit. The justification is illogical. However, I personally believe boilerplate should be omitted where possible; that's why we have a whole lot of things that "just work". Why does Python not have explicit boolification for if/while checks? REXX does (if you try to use anything else, you get a run-time error "Logical value not 0 or 1"), and that's more explicit - Python could require you to write "if bool(x)" for the case where you actually want the truthiness magic, to distinguish from "if x is not None" etc. But that's unnecessary boilerplate. Python could have required explicit nonlocal declarations for all names used in closures, but that's unhelpful too. Python strives to eliminate that kind of thing. So, my view would be: Py3-only tutorials can and probably should omit it, for the same reason that we don't advise piles of __future__ directives. You can always add stuff later for coping with Py2+Py3 execution; chances are any non-trivial code will have much bigger issues than accidentally making an old-style class. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python2.7 infinite recursion when loading pickled object
On Mon, Aug 11, 2014 at 9:40 PM, Peter Otten <__pete...@web.de> wrote: > Read again. The OP tries to delegate attribute lookup to an (existing) > attribute. > > IMO the root cause of the problem is that pickle looks up __dunder__ methods > in the instance rather than the class. The recursion comes from the attempted lookup of self.item, when __init__ hasn't been called. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Wed, Aug 13, 2014 at 11:11 PM, Isaac Morland wrote: > While I would not claim a Python sandbox is utterly impossible, I'm > suspicious that the whole "consenting adults" approach in Python is > incompatible with a sandbox. The whole idea of a sandbox is to absolutely > prevent people from doing things even if they really want to and know what > they are doing. It's certainly not *fundamentally* impossible to sandbox Python. However, the question becomes one of how much effort you're going to go to and how much you're going to restrict the code. I think I remember reading about something that's like ast.literal_eval, but allows name references; with that, plus some tiny features of assignment, you could make a fairly straight-forward evaluator that lets you work comfortably with numbers, strings, lists, dicts, etc. That could be pretty useful - but it wouldn't so much be "Python in a sandbox" as "an expression evaluator that uses a severely restricted set of Python syntax". If you start with all of Python and then start cutting out the dangerous bits, you're doomed to miss something, and your sandbox is broken. If you start with nothing and then start adding functionality, you're looking at a gigantic job before it becomes anything that you could call an applications language. So while it's theoretically possible (I think - certainly I can't say for sure that it's impossible), it's fairly impractical. I've had my own try at it, and failed quite badly (fortunately noisily and at a sufficiently early stage of development to shift). ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
On Thu, Aug 14, 2014 at 2:58 AM, Steven D'Aprano wrote: >> It's certainly not *fundamentally* impossible to sandbox Python. >> However, the question becomes one of how much effort you're going to >> go to and how much you're going to restrict the code. > > I believe that PyPy has an effective sandbox, but to what degree of > effectiveness I don't know. """ A potential attacker can have arbitrary code run in the subprocess, but cannot actually do any input/output not controlled by the outer process. Additional barriers are put to limit the amount of RAM and CPU time used. Note that this is very different from sandboxing at the Python language level, i.e. placing restrictions on what kind of Python code the attacker is allowed to run (why? read about pysandbox). """ That's quite useful, but isn't the same thing as a Python-in-Python sandbox (or even what I was doing, Python-in-C++). ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multiline with statement line continuation
On Sat, Aug 16, 2014 at 10:47 PM, Marko Rauhamaa wrote:
>
> You might be able to have it bothways. You could have:
>
>with (open(name) for name in os.listdir("config")) as files:
But that's not a tuple, it's a generator. Should generators be context
managers? Is anyone seriously suggesting this? I don't think so. Is
this solutions looking for problems?
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On Sat, Aug 23, 2014 at 6:17 AM, Glenn Linderman wrote: > "cp1251 of utf-8 encoding" is non-sensical. Either it is cp1251 or it is > utf-8, but it is not both. Maybe you meant "or" instead of "of". I'd assume "or" meant there, rather than "of", it's a common typo. Not sure why 1251, specifically, but it's not uncommon for boundary code to attempt a decode that consists of something like "attempt UTF-8 decode, and if that fails, attempt an eight-bit decode". For my MUD clients, that's pretty much required; one of the servers I frequent is completely bytes-oriented, so whatever encoding one client uses will be dutifully echoed to every other client. There are some that correctly use UTF-8, but others use whatever they feel like; and since those naughty clients are mainly on Windows, I can reasonably guess that they'll be using CP-1252. So that's what I do: UTF-8, fall-back on 1252. (It's also possible some clients will be using Latin-1, but 1252 is a superset of that.) But it's important to note that this is a method of handling junk. It's not a design intention; this is for a situation where I really want to cope with any byte stream and attempt to display it as text. And if I get something that's neither UTF-8 nor CP-1252, I will display it wrongly, and there's nothing can be done about that. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On Sat, Aug 23, 2014 at 8:26 AM, Oleg Broytman wrote: > On Sat, Aug 23, 2014 at 07:04:20AM +1000, Chris Angelico > wrote: >> On Sat, Aug 23, 2014 at 6:17 AM, Glenn Linderman >> wrote: >> > "cp1251 of utf-8 encoding" is non-sensical. Either it is cp1251 or it is >> > utf-8, but it is not both. Maybe you meant "or" instead of "of". >> >> I'd assume "or" meant there, rather than "of", it's a common typo. >> >> Not sure why 1251, specifically > >This is the encoding of Russian Windows. Files and emails in Russia > are mostly in cp1251 encoding; something like 60-70%, I think. The > second popular encoding is cp866 (Russian DOS); it's used by Windows as > OEM encoding. Yeah, that makes sense. In any case, you pick one "most likely" 8-bit encoding and go with it. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path support
On Sat, Aug 23, 2014 at 7:02 PM, Stephen J. Turnbull wrote: > Chris Barker writes: > > > So I write bytes that are encoded one way into a text file that's encoded > > another way, and expect to be abel to read that later? > > No, not you. Crap software does that. Your MUD server. Oleg's > favorite web pages with ads, or more likely the ad servers. Just to clarify: Presumably you're referring to my previous post regarding my MUD client's heuristic handling of broken encodings. It's "my server" in the sense of the one that I'm connecting to, and not in the sense that I control it. I do also run a MUD server, and it guarantees that everything it sends is UTF-8. (Incidentally, that server has the exact same set of heuristics for coping with broken encodings from other clients. There's no escaping it.) Your point is absolutely right: mess like that is to cope with the fact that there's broken stuff out there. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou wrote: > Not sure why. Just put another module named "ssl" in sys.modules directly. > You can also monkeypatch the genuine ssl module. That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Mon, Sep 1, 2014 at 11:34 PM, Antoine Pitrou wrote: > On Mon, 1 Sep 2014 23:24:39 +1000 > Chris Angelico wrote: >> On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou wrote: >> > Not sure why. Just put another module named "ssl" in sys.modules directly. >> > You can also monkeypatch the genuine ssl module. >> >> That has to be done inside the same process. But imagine this >> scenario: You have a program that gets invoked as root (or some other >> user than yourself), and you're trying to fiddle with what it sees. >> You don't have root access, but you can manipulate the file system, to >> the extent that your userid has access. What can you do to affect this >> other program? > > If you're root you shouldn't run untrusted code. See > https://docs.python.org/3/using/cmdline.html#cmdoption-I Right, which is why sslcustomize has to be controlled by that, but the possibility of patching (or monkeypatching) ssl.py isn't as big a deal. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
On Wed, Sep 3, 2014 at 12:47 PM, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > Sorry, I haven't been a very good maintainer for that buildbot (the main > > reason it never graduated to the "stable" list). If you send me your public > > SSH key, I can add it (I think - if not, I can ask Luke to do it). > > Alternatively, CentOS 6 may exhibit the same problem. > > I wonder how many of these buildbots could be maintained by the kind > of folks who show up on core-mentorship asking "how can I help?" > > Just a thought -- I wouldn't be surprised if the reaction is universal > horror and the answer is "Are you crazy? Zero! Z-E-R-O!!" > > And of course most want to write code, not sysadm. Maintaining a buildbot isn't hard. (Although one thing I'm not sure of: If my bot goes down for an extended period of time, is any sort of automated email sent to me? I don't often check their status.) But it does mean a measure of trust in some external entity, or else some very careful rules (mainly firewall), which not every coder will know about. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
On Thu, Sep 4, 2014 at 3:32 AM, francis wrote: >> does mean a measure of trust in some external entity, or else some >> very careful rules (mainly firewall), which not every coder will know >> about. > > > Just curious, is there a way to mount the infrastructure the oder way > around? One sets a system polling for sources changes, if so it starts > a build an sends per e-mail the results to some address. I'm more talking about how there's a (virtual) machine that I run, executing code sent to me by someone else (the Python devs). That machine needs a certain measure of access to the internet (to fetch code, and to run certain tests), and the easy and naive way to set it up is to give it full access to everything, which is the trust that I spoke of. Firewalling that box so it can't see anything it's not allowed to see is certainly possible, but that requires sysadmin skills, not coder skills, hence my comment. And it's very easy to make a mistake that you won't notice - everything works just fine, because normally that buildbot won't do anything it shouldn't. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] recursive closure
On Thu, Sep 11, 2014 at 12:55 PM, Li Tianqing wrote: > Hello, > Can someone explain me why gc(CPython) can not collect recursive > closure's cycle reference? There is no __del__ here, why gc can not > collect? Can you start by explaining what's not getting collected and what is? Where's the cycle you're looking at? Also, this thread would do better on python-l...@python.org rather than python-dev. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuples bug between 3.3.2 and 3.4.1
On Sun, Sep 14, 2014 at 8:13 PM, Brynjar Smári Bjarnason
wrote:
> I am using Python 3.4.1 installed with Anaconda. I tried the following
> (expecting an OrderedDict as result):
>
from collections import namedtuple
NT = namedtuple("NT",["a","b"])
nt = NT(1,2)
print(vars(nt))
> {}
>
> so the result is an empty dict. In Python 3.3.2 (downgraded in the
> same Anaconda environment) results in:
>
print(vars(nt))
> OrderedDict([('a', 1), ('b', 2)])
For what it's worth, I can't reproduce the issue on trunk CPython
(built from default branch on Aug 21, so it's a little old now), nor
on 3.4.1 as currently shipping with Debian Jessie, nor with 3.4.0 on
Windows. So this may be an Anaconda issue. Do you know if it's meant
to be a patched install of Python?
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Tue, Sep 16, 2014 at 1:34 PM, Stephen J. Turnbull wrote: > Jim J. Jewett writes: > > > In terms of best-effort, it is reasonable to treat the smuggled bytes > > as representing a character outside of your unicode repertoire > > I have to disagree. If you ever end up passing them to something that > validates or tries to reencode them without surrogateescape, BOOM! > These things are the text equivalent of IEEE NaNs. If all you know > (as in the stdlib) is that you have "generic text", the only fairly > safe things to do with them are (1) delete them, (2) substitute an > appropriate replacement character for them, (3) pass the text > containing them verbatim to other code, and (4) reencode them using > the same codec they were read with. Don't forget, these are *errors*. These are bytes that cannot be correctly decoded. That's not something that has any meaning whatsoever in text; so by definition, the only things you can do are the four you list there (as long as "codec" means both the choice of encoding and the use of the surrogateescape flag). It's like dealing with control characters when you need to print something visually, except that they have an official solution [1] and surrogateescape is unofficial. They're not real text, so you have to deal with them somehow. The bytes might each represent one character. Several of them together might represent a single character. Or maybe they don't mean anything at all, and they're just part of a chunked data format... like I was finding in the .cwk files that I was reading this weekend (it's mostly MacRoman encoding, but the text is divided into chunks separated by \0\0 and two more bytes - turns out the bytes are chunk lengths, so they don't mean any sort of characters at all). You can't know. ChrisA [1] http://www.unicode.org/charts/PDF/U2400.pdf ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Wed, Sep 17, 2014 at 1:00 AM, R. David Murray wrote: > That isn't the case in the email package. The smuggled bytes are not > errors[*], they are literally smuggled bytes. But they're not characters, which is what Stephen and I were saying - and contrary to what Jim said about treating them as characters. At best, they represent characters but in some encoding other than the one you're using, and you have no idea how many bytes form a character or anything. So you can't, for instance, word-wrap the text, because you can't know how wide these unknown bytes are, whether they represent spaces (wrap points), or newlines, or anything like that. You can't treat them as characters, so while you have them in your string, you can't treat it as a pure Unicode string - it''s a Unicode string with smuggled bytes. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Wed, Sep 17, 2014 at 3:46 AM, R. David Murray wrote: >> You can't treat them as characters, so while you have them in your >> string, you can't treat it as a pure Unicode string - it''s a Unicode >> string with smuggled bytes. > > Well, except that I do. The email header parsing algorithms all work > fine if I treat the surrogate escaped bytes as 'unknown junk' and just > parse based on the valid unicode. (Unless the header is so garbled that > it can't be parsed, of course, at which point it becomes an invalid > header). Do what, exactly? As I understand you, you treat the unknown bytes as completely opaque, not representing any characters at all. Which is what I'm saying: those are not characters. If you, instead, represented the header as a list with some str elements and some bytes, it would be just as valid (though much harder to work with); all your manipulations are done on the str parts, and the bytes just tag along for the ride. > You are right about the wrapping, though. If a header with invalid > bytes (and in this scenario we *are* talking about errors) needs to > be wrapped, we have to first decode the smuggled bytes and turn it > into an 'unknown-8bit' encoded word before we can wrap the header. Yeah, and that's going to be a bit messy. If you get 60 characters followed by 30 unknown bytes, where do you wrap it? Dare you wrap in the middle of the smuggled section? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Wed, Sep 17, 2014 at 3:55 AM, Jim Baker wrote: > Of course, if you do actually have a smuggled isolated low surrogate > FOLLOWED by a smuggled isolated high surrogate - guess what, the only > interpretation is a codepoint. Or perhaps more likely garbage. Of course it > doesn't happen so often, so maybe we are fine with the occasional bug ;) > > I personally suspect that we will resolve this by also supporting UCS-4 as a > representation in Jython 3.x for such Unicode strings, albeit with the > limitation that we have simply moved the problem to when we try to call Java > methods taking java.lang.String objects. > That'll cost efficiency, of course, but it'll guarantee correctness. And maybe, just maybe, you'll be able to put some pressure on Java itself to start supporting UCS-4 natively... One can dream. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog
On Wed, Sep 17, 2014 at 5:29 AM, R. David Murray wrote:
> Yes. I thought you were saying that one could not treat the string with
> smuggled bytes as if it were a string. (It's a string that can't be
> encoded unless you use the surrogateescape error handler, but it is
> still a string from Python's POV, which is the point of the error
> handler).
>
> Or, to put it another way, your implication was that there were no
> string operations that could be usefully applied to a string containing
> smuggled bytes, but that is not the case. (I may well have read an
> implication that was not there; if so I apologize and you can ignore the
> rest of this :)
Ahh, I see where we are getting confused. What I said was that you
can't treat the string as a *pure* Unicode string. Parts of it are
Unicode text, parts of it aren't.
> Basically, we are pretending that the each smuggled
> byte is single character for string parsing purposes...but they don't
> match any of our parsing constants. They are all "any character" matches
> in the regexes and what have you.
This is slightly iffy, as you can't be sure that one byte represents
one character, but as long as you don't much care about that, it's not
going to be an issue. I'm fairly sure you're never going to find an
encoding in which one unknown byte represents two characters, but
there are cases where it takes more than one byte to make up a
character (or the bytes are just shift codes or something). Does that
ever throw off your regexes? It wouldn't be an issue to a .* between
two character markers, but if you ever say .{5} then it might match
incorrectly.
I think we're in agreement here, just using different words. :)
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
On Thu, Sep 25, 2014 at 6:50 AM, Steve Dower wrote: > Donald Stufft wrote: >> One thing about *nix is even though you can’t write to your normal Python >> install location without root, invoking pip with permissions (assuming you >> have >> them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows >> have >> an equivalent or do you need to launch a whole new shell? > > Unfortunately not. The "easy way" is for the executable to declare that it > needs administrative privileges, and then the OS will take over and let you > approve/reject/sign-in/etc. according to your settings. When you say the executable declares this, is that a static (compile/link time) declaration, or a run-time one? That is, could pip defer the declaration until it's parsed its command line args and decided that it'll be installing into the system directory, but NOT make that declaration if it's given --user, or if it's running inside a venv, or anything else that means it doesn't need that power? If so, that could actually be a very powerful feature: a user without admin privs (or choosing not to exercise them) is still able to install into a venv, but if s/he forgets to activate the venv, the "hey, I want to modify system files" prompt will be the alarm that says the situation is not what was expected. That's what happens on my Linux system - I get errors back if I don't use sudo, but in a venv, everything works without being root. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX
On Fri, Sep 26, 2014 at 9:53 AM, Antoine Pitrou wrote:
>> In other words, os.system is *already* an attack vector, unless you only
>> use it with trusted strings. I don't think the bash env vulnerability
>> adds to the attack surface.
>>
>> Have I missed something?
>
> The part where the attack payload is passed through the environment, not
> through hypothetical user-injected command-line arguments.
Which means this also affects anything that invokes shell scripts (if
they use bash, rather than sh), even if it doesn't use os.system().
I'm just in process of checking and patching my systems (most of them
are just 'apt-get update; apt-get upgrade' followed by a quick check),
and can confirm that it does happen in Python. All you have to do is
invoke bash, either explicitly or through your target's shebang.
>>> import os, subprocess
>>> os.environ["HAHA"]="() { :;}; echo This is crafted from the environment."
>>> subprocess.call(["./test.sh"])
This is crafted from the environment.
This is from my test script.
0
>>> open("./test.sh").read()
'#!/bin/bash\necho This is from my test script.\n\n'
>>> subprocess.call(["bash","-c","echo This is from the command line."])
This is crafted from the environment.
This is from the command line.
0
But this is a bash issue, not a Python one.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Critical bash vulnerability CVE-2014-6271 may affect Python on *n*x and OSX
On Fri, Sep 26, 2014 at 10:29 AM, Devin Jeanpierre
wrote:
> As I understand it, if the attacker can help specify the environment
> (e.g. this is a CGI script), and you run os.system('echo hi'), you can
> get pwned. Even safe uses of os.system are vulnerable unless you point
> /bin/sh at a secure shell (e.g. patched bash).
/bin/sh may well not point to bash anyway - it doesn't on any of my
systems. Debian provides dash instead, much faster than bash. But if
you're invoking a script that calls for bash, then it's vulnerable.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.4.2rc1 online docs: cannot access tkinter, idle pages
On Mon, Oct 6, 2014 at 6:49 AM, Terry Reedy wrote: > I can access every page I have tried except for the tkinter and idle pages, > which either give interminable 'Connecting...' or in one try, 404. > https://docs.python.org/3/library/tkinter.html > https://docs.python.org/3/library/idle.html Is this still the case? I just tried them and they work for me. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing 2.7.x
On Tue, Oct 7, 2014 at 4:33 AM, Skip Montanaro wrote: > On Mon, Oct 6, 2014 at 12:24 PM, Ned Deily wrote: >> So 2.7.x is not "security only" and wouldn't reach that stage until 2020 >> under current policy. > > Apparently no other 2.x release qualifies as "security only" at this > point? I would have expected at least 2.6 to fall into that category. Was until about a year ago. http://legacy.python.org/dev/peps/pep-0361/ ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] mUTF-7 support?
On Fri, Oct 10, 2014 at 11:52 AM, Jesus Cea wrote: > Actually, it doesn't work in Python 2 either. It never supported > international mailbox names. > > Should I dare to suggest to port this to 2.7, since 2.7 is special and > will be supported for a long time?. Or maybe this is something like > "Yes, Python 2 is broken, the real deal is Python 3"? :). That's ultimately up to the release manager, but IMO this sounds like a bug to be fixed, more than a feature being added. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] mUTF-7 support?
On Fri, Oct 10, 2014 at 3:05 PM, Glenn Linderman wrote: > There are still lots of idiotic web sites that assume everything in front of > the @ must be a letter, digit, dot, or hyphen, and even some that only > permit one dot after the @... even though for 30 years or so, the RFCs have > permitted a nice variety of other special characters, although not all of > them. And heaps that require a dot after the @, which is definitely not a requirement. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
On Sat, Oct 11, 2014 at 2:45 AM, Jesus Cea wrote: > On 03/09/14 02:37, Antoine Pitrou wrote: > >> I'm not sure that's an answer to the problem. What we need is not more >> machines, but dedicated buildbot maintainers. > > I would love to get an email if my buildbots are consistently RED for a > few hours. > > In the past Antoine, Victor and others pinged me about issues in my > buildbots (OpenIndiana). I feel shame and I think it is not very > productive either. Likewise. In the earliest days of my buildbots, I kept a very close eye on them (which was good; they had some occasional network glitches, which I had to resolve), but once they became stable, I stopped watching. Now, I almost never check them - as long as the VM's running, I basically assume all's well. It'd be very helpful if a computer could auto-email any time they're down for any length of time. Is the info available in some way? Could I write a little monitor at my end that asks every hour if my buildbots can be seen? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Sun, Oct 26, 2014 at 7:50 AM, Steve Dower wrote: > Ray Donnelly wrote: >> What is it that you >> are afraid of if CPython can be compiled out of the box using >> mingw/MinGW-w64? Why are you fighting so hard against having option. > > I'm afraid of users having numpy crash because they're using an MSVC CPython > instead of a mingw CPython. I'm afraid of users not being able to use library > A and library B at the same time because A requires MSVC CPython and B > requires mingw CPython. (I can produce more examples if you like, but the > general concern is having a fragmented community, as I said in my previous > post.) > It might fragment the community to have multiple different binary distributions. But it ought to be possible for any person/organization to say "We're going to make our own build of Python, with these extension modules, built with this compiler, targeting this platform", and do everything from source. That might mean they can no longer take the short-cut of "download someone's MSVC-built extension and use it as-is", but they should be able to take anyone's extension and build it on their chosen compiler. Having MinGW as a formally supported platform would make life a lot easier for people who want to test CPython patches, for instance - my building and testing of PEP 463-enhanced Python was Linux-only, because I didn't want to try to set up an entire new buildchain just to try to get a Windows binary going. There's absolutely no need for that to be binary-compatible with anything else; as long as it'll run the standard library, it'll do. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Sun, Oct 26, 2014 at 8:47 AM, Antoine Pitrou wrote: > And how do you know that it would have worked with MSVC if you only use > MinGW? > If you want to ensure compatibility with MSVC, you must build with MSVC. > There's no working around that. Precisely. If you build with MinGW, you can't ensure compatibility with MSVC. Reread my post: I gave two examples of situations where that isn't a problem. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Sun, Oct 26, 2014 at 8:59 AM, Antoine Pitrou wrote: > How do you know this isn't a problem, since you haven't *tested* with > MSVC? > Why on Earth would you want to test your PEP work with an unsupported > Windows compiler and runtime, rather than with the officially supported > compiler and runtime? This discussion revolved around supporting MinGW in addition to MSVC. If it had been supported when I was doing that, I could have spun myself up a Windows build and tested it. Since it was (and so far still is) not, the hassle of hunting down a valid MSVC that could build for Win XP (as that's what my test box runs) was simply not worthwhile. My point is that there is no community fragmentation happening here; the only fragmentation is of binary distribution of extension modules, and there are several ways in which this needn't be a problem. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou wrote: > My point is that your "Windows build" would not have the same behaviour > as a MSVC-produced Windows build, and so testing it with it would not > certify that your code would actually be compatible with genuine > MSVC builds of CPython, which we will not stop supporting. > So you're saying it's impossible to support two compilers? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Impact of Windows PowerShell OneGet ?
On Thu, Oct 30, 2014 at 6:34 AM, Glenn Linderman wrote: > (I'm actually not sure if *nix package managers allow multiple repositories > or not, but from the way people talk about them, it always sounds like a > "distribution" also provides "a repository" of additional packages). I'm fairly sure they all do. Certainly with apt (the Debian package manager), it's common to add additional repositories; for instance, PostgreSQL can be obtained either from the default Debian repos or from Postgres's own hosting (which usually has more versions available). A distribution will always provide a repository, and there are plenty of distros that provide only a small repo and chain to upstream for most packages - for instance AntiX has its own, and then pulls in debian.org and a few others. Adding a local-network repo is fairly straight-forward. Most likely, OneGet won't replace pip/PyPI, any more than apt or yum does; but it may be worth having Python itself available that way. That might simply mean having someone package up Python and put it on an appropriate server, or maybe python.org could end up hosting a repo. I've no idea what "trusted" will mean; in the case of apt, any sysadmin can deem any repo to be trusted (by importing its key), but this might be more along the lines of "only curated packages" or something. To what extent will Windows 10 users expect all their software to come via OneGet? That's the question. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] py34 makes it harder to read all of a pty
On Thu, Nov 13, 2014 at 6:34 PM, Charles-François Natali wrote: > 2014-11-12 22:16 GMT+00:00 Buck Golemon : >> This is due to the fix for issue21090, which aimed to un-silence errors >> which previously went unheard. The fix is for me, as a user, to write a loop >> that uses os.read and interpretes EIO as EOF. This is what I had hoped >> file.read() would do for me, however, and what it used to do in previous >> pythons. > > > There's no reason for read() to interpret EIO as EOF in the general > case: it was masked in previous versions because of a mere bug. The > behavior is now correct, although being able to retrieve the data read > so far in case of a buffered read could be useful. Every change breaks someone's workflow. http://xkcd.com/1172/ ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OneGet provider for Python
On Sun, Nov 16, 2014 at 2:40 AM, Paul Moore wrote: > On 15 November 2014 15:17, Benjamin Peterson wrote: >> On Sat, Nov 15, 2014, at 05:54, Nathaniel Smith wrote: >>> On 15 Nov 2014 10:10, "Paul Moore" wrote: >>> > >>> > > Incidentally, it would be really useful if python.org provided stable >>> > > url's that always redirected to the latest .msi installers, for >>> > > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on >>> > > scraping the web site) for this. >>> > >>> > https://www.python.org/ftp/python/$ver/python-$ver.msi >>> > https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi >>> >>> Right, but what's the URL for "the latest 2.7.x release" or "the latest >>> 3.x.x release"? >> >> The website has an API you know. > > Um, no. Where can I find out about it? I didn't know either, and I've been pointing people to the .msi installers periodically - just by manually hunting down the latest link and recommending it. If there's an easy way to define the two tokens Nathaniel described, I'd like to hear it too. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 20, 2014 at 7:48 AM, MRAB wrote:
> The PEP says """any generator that depends on an implicitly-raised
> StopIteration to terminate it will have to be rewritten to either catch
> that exception or use a for-loop"""
>
> Shouldn't that be "... explicitly-raised ...", because returning raises
> StopIteration implicitly? ("raise StopIteration" is explicit)
The point here is primarily about some other function (maybe a
next(iter), or maybe something else entirely) raising StopIteration.
(If it explicitly raises StopIteration right there in the generator,
it can be trivially converted into a return statement, anyway.) The
return statement is an explicit indication that the generator should
now return; permitting a StopIteration to bubble up through and out is
the implicit option; but the 'plicitness' isn't necessarily obvious.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 20, 2014 at 3:45 AM, Nick Coghlan wrote:
> The part I found most compelling was when you pointed out that in the
> special method implementations, the normal return path was always spelled
> with "return", while the "value missing" result was indicated with a special
> kind of exception (StopIteration, AttributeError, IndexError or KeyError),
> and then any other exception was consider unexpected.
>
> Generators add the third notion of being able to suspend execution via
> "yield", which then left them with two different ways of spelling
> termination inside the frame: "return" OR "raise StopIteration". The second
> spelling ("raise StopIteration") is then inherently surprising, as it's
> entirely redundant, *except* in that it allows you to effectively have a
> "hidden return" in a generator frame that can't be done anywhere else.
(The above was said on -ideas, but discussion is now moving to -dev,
so I hope it's okay to send the response there.)
Added a paragraph to the PEP draft:
https://github.com/Rosuav/GenStopIter/commit/695961
Should today's date be added to the Post-History?
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 6:36 AM, Guido van Rossum wrote: > It would also be useful if we could extend the PEP with some examples of the > various categories of fixes that can be applied easily, e.g. a few examples > of "raise StopIteration" directly in a generator that can be replaced with > "return" (or omitted, if it's at the end); a few examples of situations > where "yield from" can supply an elegant fix (and an alternative for code > that needs to be backward compatible with Python 3.2 or 2.7); and finally > (to be honest) an example of code that will require being made more > complicated. > > Oh, and it would also be nice if the PEP included some suggested words that > 3rd party educators can use to explain the relationship between > StopIteration and generators in a healthier way (preferably a way that also > applies to older versions). > > Chris, are you up to drafting these additions? Sure, no problem. Will knock something up shortly. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 9:04 AM, Guido van Rossum wrote: > On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka > wrote: >> >> On 20.11.14 21:58, Antoine Pitrou wrote: >>> >>> To me "generator_return" sounds like the addition to generator syntax >>> allowing for return statements (which was done as part of the "yield >>> from" PEP). How about "generate_escape"? >> >> >> Or may be "generator_stop_iteration"? > > > Or just "generator_stop"? Unrelated to the GeneratorExit exception. I don't think that'll be a problem though. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 10:34 AM, Chris Angelico wrote: > On Fri, Nov 21, 2014 at 6:36 AM, Guido van Rossum wrote: >> It would also be useful if we could extend the PEP with some examples of the >> various categories of fixes that can be applied easily, e.g. a few examples >> of "raise StopIteration" directly in a generator that can be replaced with >> "return" (or omitted, if it's at the end); a few examples of situations >> where "yield from" can supply an elegant fix (and an alternative for code >> that needs to be backward compatible with Python 3.2 or 2.7); and finally >> (to be honest) an example of code that will require being made more >> complicated. >> >> Oh, and it would also be nice if the PEP included some suggested words that >> 3rd party educators can use to explain the relationship between >> StopIteration and generators in a healthier way (preferably a way that also >> applies to older versions). >> >> Chris, are you up to drafting these additions? > > Sure, no problem. Will knock something up shortly. Examples and explanatory text added. https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sat, Nov 22, 2014 at 12:47 AM, Raymond Hettinger wrote: > Also, the proposal breaks a reasonably useful pattern of calling > next(subiterator) inside a generator and letting the generator terminate > when the data stream ends. Here is an example that I have taught for > years: > > def izip(iterable1, iterable2): > it1 = iter(iterable1) > it2 = iter(iterable2) > while True: > v1 = next(it1) > v2 = next(it2) > yield v1, v2 Is it obvious to every user that this will consume an element from it1, then silently terminate if it2 no longer has any content? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sat, Nov 22, 2014 at 2:58 AM, Steven D'Aprano wrote: > Since zip() is documented as > halting on the shorter argument, it can't raise an exception. So what > other options are there apart from silently consuming the value? Sure, it's documented as doing that. But imagine something that isn't a well-known function - all you have is someone writing a generator that calls next() in multiple places. Is it obvious that it it'll silently terminate as soon as any one of those iterators is exhausted? In many cases, an exception (probably ValueError?) would be the most obvious response. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sat, Nov 22, 2014 at 3:37 AM, Donald Stufft wrote: > I don’t have an opinion on whether it’s enough of a big deal to actually > change > it, but I do find wrapping it with a try: except block and returning easier > to understand. If you showed me the current code unless I really thought about > it I wouldn't think about the fact that the next() calls can cause the > generator to terminate. And don't forget, by the way, that you can always request the current behaviour by explicitly wrapping the body of the function in try/except: def izip(iterable1, iterable2): try: it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 except StopIteration: pass That's exactly what current behaviour does, and if you think that that try block is too broad and should be narrowed, then you should support this proposal :) ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 6:49 AM, Ron Adam wrote: > > OPTION 1: > > Make comprehensions act more like generator expressions. > > It would mean a while loop in the object creation point is converted to a > for loop. (or something equivalent.) > > Then both a comprehension and a generator expressions can be viewed as > defining iterators, with the same behaviour, rather than comprehensions > defining the body of the loop, which has the different but valid behaviour > of StopIteration escaping. > > This would make explaining them a *lot* easier as they become the same thing > used in a different context, rather than two different things used in what > appears to be similar contexts. > > > I think this fits with what Guido wants, but does so in a narrower scope, > only effecting comprehensions. StopIteration is less likely to leak out. A list comp is usually compared to a statement-form loop. def stop(): raise StopIteration lst = [stop() for x in (1,2,3)] lst = [] for x in (1,2,3): lst.append(stop()) At the moment, these are identical (virtually; the pedantic will point out that 'x' doesn't leak out of the comprehension) - in each case, the exception raised by the body will bubble up and be printed to the console. But a generator expression is different: lst = list(stop() for x in (1,2,3)) In this form, lst is an empty list, and nothing is printed to the console. Making comprehensions work more like generator expressions would, IMO, imply making the same change to all for loops: having a StopIteration raised by the body of the loop quietly terminate the loop. This is backwards. Most of Python is about catching exceptions as narrowly as possible: you make your "try" blocks small, you use specific exception types rather than bare "except:" clauses, etc, etc, etc. You do your best to catch ONLY those exceptions that you truly understand, unless you're writing a "catch, log, and reraise" or "catch, log, and go back for more work" generic handler. A 'for' loop currently is written more-or-less like this: for var in expr: body it = iter(expr) while True: try: var = next(it) except StopIteration: break body But this suggestion of yours would make it become: it = iter(expr) while True: try: var = next(it) body except StopIteration: break I believe this to be the wrong direction to make the change. Instead, generator expressions should be the ones to change, because that's a narrowing of exception-catching scope. Currently, every generator function is implicitly guarded by "try: .. except StopIteration: pass". This is allowing errors to pass silently, to allow a hack involving non-local control flow (letting a chained function call terminate a generator) - violating the Zen of Python. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam wrote: >> Making comprehensions work more like generator expressions >> would, IMO, imply making the same change to all for loops: having a >> StopIteration raised by the body of the loop quietly terminate the >> loop. > > > I'm not suggesting making any changes to generator expressions or for loops > at all. They would continue to work like they currently do. But if you're suggesting making list comps react to StopIteration raised by their primary expressions, then to maintain the correspondence between a list comp and its expanded form, for loops would have to change too. Or should that correspondence be broken, in that single-expression loop constructs become semantically different from statement loop constructs? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 7:30 AM, Raymond Hettinger wrote: > Legitimate Use Cases for Raising StopIteration in a Generator > > > In a producer/consumer generator chain, the input generator signals > it is done by raising StopIteration and the output generator signals > that it is done by raising StopIteration (this isn't in dispute). > > That use case is currently implemented something like this: > > def middleware_generator(source_generator): > it = source_generator() > input_value = next(it) > output_value = do_something_interesting(input_value) > yield output_value Does your middleware_generator work with just a single element, yielding either one output value or none? Or is it more likely to be iterating over the source generator: def middleware_generator(source_generator): for input_value in source_generator: yield do_something_interesting(input_value) MUCH tidier code, plus it's safe against unexpected StopIterations. Even if you can't do it this way, all you need is to stick a 'break' at the end of the loop, if the try/except is so abhorrent. Wouldn't be the first time I've seen a loop with a hard break at the end of it. > This doesn't make the code better in any way. The new code > is wordy, slow, and unnecessarily convoluted: > > def middleware_generator(source_generator): > it = source_generator() > try: > input_value = next(it) > except StopIteration: > return # This causes StopIteration to be reraised > output_value = do_something_interesting(input_value) > yield output_value The raising of StopIteration here is an implementation detail; you might just as well have a comment saying "This causes the function to set an exception state and return NULL", which is what happens at the C level. What happens if do_something_interesting happens to raise StopIteration? Will you be surprised that this appears identical to the source generator yielding nothing? That's current behaviour. You don't have the option of narrowing the try/except scope as you've done above. > Is next() Surprising? > - > > If someone in this thread says that they were suprised that next() could > raise StopIteration, I don't buy it. Agreed, I don't think that's surprising to anyone. > Being able to consume a value from an iterator stream is a fundamental > skill and not hard to learn (when I teach iterators and generators, the > operation of next() has never been a stumbling block). In anything other than a generator, you're expected to cope with two possible results from next(): a returned value or a raised StopIteration. Suppose you want to read a file with a header - you'd need to do something like this: def process_file(f): f = iter(f) try: header=next(f) except StopIteration: cope_somehow_maybe_return for line in f: process_line_with_headers(line, header) Currently, *if and only if* you're writing a generator, you have an implicit "except StopIteration: return" there. Anywhere else, you need to catch that exception. Iterators are an implementation detail of generators. There is no particular reason for a generator author to be aware of iterator protocol, any more than this class needs to be: class X: def __iter__(self): return iter([1,2,3,4]) It's perfectly iterable, just as a generator is, and it knows nothing about StopIteration. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 11:05 AM, Ron Adam wrote:
> Se we have these...
>
> Tuple Comprehension (...)
> List Comprehension [...]
> Dict Comprehension {...} Colon make's it different from sets.
> Set Comprehension {...}
>
> I don't think there is any other way to create them. And they don't actually
> expand to any python code that is visible to a programmer. They are self
> contained literal expressions for creating these few objects.
Hmmm, there's no such thing as tuple comprehensions. Are you confusing
comprehension syntax (which has a 'for' loop in it) with
tuple/list/etc display, which doesn't?
lst = [1,2,3,4] # not a comprehension
even = [n*2 for n in lst] # comprehension
> Here is what I think(?) list comps do currently.
>
> lst = [expr for items in itr if cond]
>
> Is the same as.
>
> lst = []
> for x in itr: # StopIteration caught here.
> if cond: # StopIteration bubbles here.
> lst.append(expr) # StopIteration bubbles here.
Pretty much. It's done in a nested function (so x doesn't leak), but
otherwise, yes.
> And it would be changed to this...
>
> def comp_expression():
> for x in itr: # StopIteration caught here.
> if cond:
>list.append(expr)
>
> # StopIteration from cond and expr caught here.
> lst = list(x for x in comp_expression())
That wouldn't quite work, but this would:
def ():
ret = []
try:
for x in itr:
if cond:
ret.append(x)
except StopIteration:
pass
return ret
lst = ()
(And yes, the name's syntactically illegal, but if you look at a
traceback, that's what is used.)
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 11:51 AM, Ron Adam wrote: > > > On 11/22/2014 06:20 PM, Chris Angelico wrote: >> >> Hmmm, there's no such thing as tuple comprehensions. > > Just didn't think it through quite well enough. But you are correct, that > would be a generator expression. > > One less case to worry about. :-) Ah right, no probs. >>> >And it would be changed to this... >>> > >>> > def comp_expression(): >>> > for x in itr: # StopIteration caught here. >>> > if cond: >>> >list.append(expr) >>> > >>> > # StopIteration from cond and expr caught here. >>> > lst = list(x for x in comp_expression()) > > Right, the list.append() should be a yield(expr). In that case, your second generator expression is entirely redundant; all you want is list(comp_expression()). But the example doesn't say *why* this version should terminate on a StopIteration raised by expr, when the statement form would print an exception traceback. > The real question is how much breakage would such a change make? That will > probably need a patch in order to test it out. There's one attached here: http://bugs.python.org/issue22906 It doesn't create a __future__ directive, just applies the change globally. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 12:11 PM, Raymond Hettinger
wrote:
> The worry is that your proposal intentionally breaks that code which is
> currently
> bug free, clean, fast, stable, and relying on a part of the API that has
> been
> guaranteed and documented from day one.
(I'd just like to mention that this isn't "my proposal", beyond that
I'm doing the summarizing and PEP writing. The proposal itself is
primarily derived from one of Guido's posts on -ideas.)
> Here's one from Fredrick Lundh's ElementTree code in the standard library
> (there are several other examples besides this one in his code are well):
>
> def iterfind(elem, path, namespaces=None):
> # compile selector pattern
> cache_key = (path, None if namespaces is None
> else tuple(sorted(namespaces.items(
> if path[-1:] == "/":
> path = path + "*" # implicit all (FIXME: keep this?)
> try:
> selector = _cache[cache_key]
> except KeyError:
> if len(_cache) > 100:
> _cache.clear()
> if path[:1] == "/":
> raise SyntaxError("cannot use absolute path on element")
> next = iter(xpath_tokenizer(path, namespaces)).__next__
> token = next()
> selector = []
> while 1:
> try:
> selector.append(ops[token[0]](next, token))
> except StopIteration:
> raise SyntaxError("invalid path")
> try:
> token = next()
> if token[0] == "/":
> token = next()
> except StopIteration:
> break
> _cache[cache_key] = selector
> # execute selector pattern
> result = [elem]
> context = _SelectorContext(elem)
> for select in selector:
> result = select(context, result)
> return result
Most of the next() calls are already guarded with try/except; from
what I can see, only the first one isn't. So this wouldn't be much of
a change here.
> And here is an example from the pure python version of one of the itertools:
>
> def accumulate(iterable, func=operator.add):
> 'Return running totals'
> # accumulate([1,2,3,4,5]) --> 1 3 6 10 15
> # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120
> it = iter(iterable)
> total = next(it)
> yield total
> for element in it:
> total = func(total, element)
> yield total
Again, only the first one needs protection, and all that happens is
that there's clearly a control flow possibility here (that the first
"yield total" might not be reached). Currently, *any* function call
has the potential to be a silent control flow event.
> And here is an example from Django:
>
> def _generator():
> it = iter(text.split(' '))
> word = next(it)
> yield word
> pos = len(word) - word.rfind('\n') - 1
> for word in it:
> if "\n" in word:
> lines = word.split('\n')
> else:
> lines = (word,)
> pos += len(lines[0]) + 1
> if pos > width:
> yield '\n'
> pos = len(lines[-1])
> else:
> yield ' '
> if len(lines) > 1:
> pos = len(lines[-1])
> yield word
> return ''.join(_generator())
When you split a string, you're guaranteed at least one result, ergo
'it' is guaranteed to yield at least one word. So this one wouldn't
need to be changed - it can't possibly raise RuntimeError.
> I could scan for even more examples, but I think you get the gist.
> All I'm asking is that you consider that your proposal will do more
> harm than good. It doesn't add any new capability at all.
> It just kills some code that currently works.
I have considered it, and I'm not convinced that it will. I see lots
of people saying "code will have to be changed", but that's exactly
the same concern that people raise about switching from the sloppy Py2
merging of text and bytes to the strict Py3 separation - yes, code has
to be changed, but it's definitely much better to have immediate and
obvious failures when something's wrong than to have subtle behavioral
changes.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 4:59 PM, Nick Coghlan wrote: > The learning curve on git is still awful - it offers no compelling > advantages over hg, and GitHub doesn't offer any huge benefits over > BitBucket for Sphinx based documentation (ReadTheDocs works just as > well with either service). The learning curve for source control as a concept is pretty steep. Once someone's learned one DVCS, learning another is much easier, and I don't know that it makes a lot of difference whether you learn git and then hg, or hg and then git. I learned git first, and mastered hg basics by keeping a Rosetta Stone chart handy; given that the document I was reading was intended for the reverse case, I expect it's not too hard to get the basics of git once you know hg. Just as git offers few advantages over hg, hg offers few advantages over git. If git and GitHub are where most people are, I would support using them for Python. I'm one of those PEP draft authors who starts with his own repo on GitHub and sends drafts in, and Guido had to remind me that I can simply test my edits in the peps repo before posting them (to make sure I haven't mucked up the markup); if the peps repo were itself hosted on GitHub, or at least in a git repo, I could have a workflow that directly incorporates that, instead of being "off to the side" with periodic manual imports. That said, it does make sense for CPython *itself* to use Mercurial, simply because it's written in Python. I don't know how strong that philosophical argument is with people, but I wouldn't argue against it. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Sun, Nov 23, 2014 at 5:03 PM, Wes Turner wrote: > hg imuutability is certainly a primarily attractive feature; > along with the keyring support. What exactly do you mean by immutability? Are you talking about how git allows a "force push" that can destroy data? That can be rejected in a repo's hook scripts; also, I'm fairly sure I remember reading somewhere about how to do that with hg, too. It's all bits of data inside computers; nothing's immutable. Both DVCSes pledge to make it obvious any time something is altered, not to make it impossible to alter. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 2:11 AM, Ethan Furman wrote: > On 11/22/2014 08:53 PM, Guido van Rossum wrote: >> >> In order to save everyone's breath, I am *accepting* the proposal of PEP >> 479. > > Excellent. > > Chris, thank you for your time, effort, and thoroughness in championing this > PEP. > Thank you, it's nice to have a successful one to counterbalance the "failure" of PEP 463. (Which, incidentally, never actually got a resolution. It's still showing up as 'Draft' status.) I'm actually quite impressed with how on-topic the discussion threads went. They didn't ramble nearly as much as I thought they would. Thank you to all who participated, contributed suggestions, complained about the existing text, and generally worked hard to make sure I had more than enough material to draw on! Particular thanks to Guido, pushing changes through and being patient with my mistakes in markup, and adding content directly to the document. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 5:28 AM, Ron Adam wrote:
> With the passage of the PEP, it will change what is different about them
> once it's in full effect. The stop hack won't work in both, and you may get
> a RuntimeError in generator expressions where you would get StopIteration in
> list-comps. (Is this correct?)
"them" being list comps and generator expressions?
The stop hack won't work in either (currently it does work in
genexps), but you'd get a different exception type if you attempt it.
This is correct. It's broadly similar to this distinction:
>>> {1:2,3:4}[50]
Traceback (most recent call last):
File "", line 1, in
KeyError: 50
>>> [1,2,3,4][50]
Traceback (most recent call last):
File "", line 1, in
IndexError: list index out of range
In both lists and dicts, you can't look up something that isn't there.
But you get a slightly different exception type (granted, these two do
have a common superclass) depending on the exact context. But the
behaviour will be effectively the same.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Mon, Nov 24, 2014 at 7:18 AM, Mark Shannon wrote:
> Hi,
>
> I have serious concerns about this PEP, and would ask you to reconsider it.
Hoping I'm not out of line in responding here, as PEP author. Some of
your concerns (eg "5 days is too short") are clearly for Guido, not
me, but perhaps I can respond to the rest of it.
> [ Very short summary:
> Generators are not the problem. It is the naive use of next() in an
> iterator that is the problem. (Note that all the examples involve calls to
> next()).
> Change next() rather than fiddling with generators.
> ]
>
> StopIteration is not a normal exception, indicating a problem, rather it
> exists to signal exhaustion of an iterator.
> However, next() raises StopIteration for an exhausted iterator, which really
> is an error.
> Any iterator code (generator or __next__ method) that calls next() treats
> the StopIteration as a normal exception and propogates it.
> The controlling loop then interprets StopIteration as a signal to stop and
> thus stops.
> *The problem is the implicit shift from signal to error and back to signal.*
The situation is this: Both __next__ and next() need the capability to
return literally any object at all. (I raised a hypothetical
possibility of some sort of sentinel object, but for such a sentinel
to be useful, it will need to have a name, which means that *by
definition* that object would have to come up when iterating over the
.values() of some namespace.) They both also need to be able to
indicate a lack of return value. This means that either they return a
(success, value) tuple, or they have some other means of signalling
exhaustion.
I'm not sure what you mean by your "However" above. In both __next__
and next(), this is a signal; it becomes an error as soon as you call
next() and don't cope adequately with the signal, just as KeyError is
an error.
> 2. The proposed solution does not address this issue at all, but rather
> legislates against generators raising StopIteration.
Because that's the place where a StopIteration will cause a silent
behavioral change, instead of cheerily bubbling up to top-level and
printing a traceback.
> 3. Generators and the iterator protocol were introduced in Python 2.2, 13
> years ago.
> For all of that time the iterator protocol has been defined by the
> __iter__(), next()/__next__() methods and the use of StopIteration to
> terminate iteration.
>
> Generators are a way to write iterators without the clunkiness of explicit
> __iter__() and next()/__next__() methods, but have always obeyed the same
> protocol as all other iterators. This has allowed code to rewritten from one
> form to the other whenever desired.
>
> Do not forget that despite the addition of the send() and throw() methods
> and their secondary role as coroutines, generators have primarily always
> been a clean and elegant way of writing iterators.
This question has been raised several times; there is a distinct
difference between __iter__() and __next__(), and it is only the
latter which is aware of StopIteration. Compare these three classes:
class X:
def __init__(self): self.state=0
def __iter__(self): return self
def __next__(self):
if self.state == 3: raise StopIteration
self.state += 1
return self.state
class Y:
def __iter__(self):
return iter([1,2,3])
class Z:
def __iter__(self):
yield 1
yield 2
yield 3
Note how just one of these classes uses StopIteration, and yet all
three are iterable, yielding the same results. Neither Y nor Z is
breaking iterator protocol - but neither of them is writing an
iterator, either.
> 4. Porting from Python 2 to Python 3 seems to be hard enough already.
Most of the code broken by this change can be fixed by a mechanical
replacement of "raise StopIteration" with "return"; the rest need to
be checked to see if they're buggy or unclear. There is an edge case
with "return some_value" vs "raise StopIteration(some_value)" (the
former's not compatible with 2.7), but apart from that, the
recommended form of code for 3.7 will work in all versions of Python
since 2.2.
> 5. I think I've already covered this in the other points, but to reiterate
> (excuse the pun):
> Calling next() on an exhausted iterator is, I would suggest, a logical
> error.
How do you know that it's exhausted, other than by calling next() on it?
> It also worth noting that calling next() is the only place a StopIteration
> exception is likely to occur outside of the iterator protocol.
This I agree with.
> An example
> --
>
> Consider a function to return the value from a set with a single member.
> def value_from_singleton(s):
> if len(s) < 2: #Intentional error here (should be len(s) == 1)
>return next(iter(s))
> raise ValueError("Not a singleton")
>
> Now suppose we pass an empty set to value_from_singleton(s), then we get a
> StopIteration exception, which is a bit weird, but not too bad.
Only a little we
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 10:18 AM, Ron Adam wrote:
>> The stop hack won't work in either (currently it does work in
>> genexps), but you'd get a different exception type if you attempt it.
>> This is correct. It's broadly similar to this distinction:
>>
> >>>{1:2,3:4}[50]
>>
>> Traceback (most recent call last):
>>File "", line 1, in
>> KeyError: 50
>
> >>>[1,2,3,4][50]
>>
>> Traceback (most recent call last):
>>File "", line 1, in
>> IndexError: list index out of range
>
>
> Comprehensions insert/append items, so you wouldn't get those unless you are
> reading from another object and would bubble out of generators too. Which is
> good because it's most likely an error that needs fixing.
My point is that doing the same errant operation on a list or a dict
will give different exceptions. In the same way, calling next() on an
empty iterator will raise StopIteration normally, but might raise
RuntimeError instead. It's still an exception, it still indicates a
place where code needs to be changed (unlike, say, a ValueError when
doing type conversions, which often means *data* needs to be changed),
so it doesn't hugely matter _which_ exception is raised. Also, I'm
hoping that someone can improve on my patch by having the full
traceback from the original StopException become visible - then, the
RuntimeError's __context__ will give all the info you would want.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?
On Mon, Nov 24, 2014 at 2:42 PM, Guido van Rossum wrote: > Then there's this. http://git-man-page-generator.lokaltog.net/ Wow scarily accurate. http://git-man-page-generator.lokaltog.net/#2d1a13476a5f32c4db27fd7aa89a84f3 Anything to do with git submodules is virtually impossible to distinguish from an elaborate practical joke. I know this because I have tried to use them. But there is hope: the git maintainers *do* accept docs patches. https://www.kernel.org/pub/software/scm/git/docs/git-config.html#_configuration_file Formerly, this said "You will find a description of non-core porcelain configuration variables in the respective porcelain documentation in the appropriate manual page.". Did you know that that means it's acceptable for a third-party git hook to make use of 'git config' to allow end users to configure it? Neither did I, till I asked on the mailing list. However, there are enough git front-ends and tutorials that mean you can generally ignore the man pages until you need something more complicated. Basically, consider the git man pages at the same time you'd consider enabling a Mercurial extension. That's how it seems to be. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 26, 2014 at 2:20 AM, Steven D'Aprano wrote:
> I wouldn't interpret it like that.
>
> Calling next() on an empty iterator raises StopIteration. That's not a
> bug indicating a failure, it's the protocol working as expected. Your
> response to that may be to catch the StopIteration and ignore it, or to
> allow it to bubble up for something else to deal with it. Either way,
> next() raising StopIteration is not a bug, it is normal behaviour.
>
> (Failure to deal with any such StopIteration may be a bug.)
>
> However, if next() raises RuntimeError, that's not part of the protocol
> for iterators, so it is almost certainly a bug to be fixed. (Probably
> coming from an explicit "raise StopIteration" inside a generator
> function.) Your fix for the bug may be to refuse to fix it and just
> catch the exception and ignore it, but that's kind of nasty and hackish
> and shouldn't be considered good code.
>
> Do you agree this is a reasonable way to look at it?
Yes. Specifically, your parenthesis in the middle is the important
bit. If you have a csv.DictReader, KeyError might be an important part
of your protocol (maybe you have an optional column in the CSV file),
but it should be caught before it crosses the boundary of "part of
your protocol". At some point, it needs to be converted into
ValueError, perhaps, or replaced with a default value, or some other
coping mechanism is used. Failure to deal with StopIteration when
calling next() is failure to cope with all of that function's
protocol, and that is most likely to be a bug. (There are times, and
some of them have been mentioned in these discussion threads, where
calling next() can never raise StopIteration, so there need be no
try/except - eg it=iter(string.split(" ")) - but that just means that
a StopIteration from that call is an error somewhere else. I'm
definitely happy for that kind of "shouldn't happen" to turn into a
RuntimeError rather than being left as an unexpectedly-short
generator.)
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher wrote: > Yield can also raise StopIteration, if it's thrown in. The current > interaction of generator.throw(StopIteration) with yield from can't be > emulated under the PEP's behavior, though it's not clear that that's a > problem. > Hrm. I have *absolutely* no idea when you would use that, and how you'd go about reworking it to fit this proposal. Do you have any example code (production or synthetic) which throws StopIteration into a generator? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Wed, Nov 26, 2014 at 11:58 AM, Greg wrote: > The Abstract claims that the proposal will "unify the behaviour of > list comprehensions and generator expressions", but it doesn't do > that. I don't know that it completely unifies the behaviours, but it does merge them on the specific situation of a leaking StopIteration. With the original code examples that sparked this all off (see the first footnote in the PEP), current CPython has the list-comp form terminate cleanly, while the genexp form infinitely loops. With the POC patch on the issue tracker, both forms cause RuntimeError. Is there a better word than "unify" for that? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Wed, Nov 26, 2014 at 10:24 PM, Nick Coghlan wrote: > The other key aspect is that it changes the answer to the question > "How do I gracefully terminate a generator function?". The existing > behaviour has an "or" in the answer: "return from the generator frame, > OR raise StopIteration from the generator frame". That then leads to > the follow on question: "When should I use one over the other?". > > The "from __future__ import generator_stop" answer drops the "or", so > it's just: "return from the generator frame". If I understand you correctly, you agree that this is a benefit, correct? > The key downside is that it means relatively idiomatic code like: > > def my_generator(): > ... > yield next(it) > ... > > Now needs to be written out explicitly as: > > def my_generator(): > ... >try: > yield next(it) > except StopIteration > return > ... > > That's not especially easy to read, and it's also going to be very > slow when working with generator based producer/consumer pipelines. I'm not sure how often the ease-of-reading concern will come up, but I can at least benchmark the performance of it. I have two virtually-identical builds of CPython 3.5, one with and one without the POC patch. There's no guarantee that this will properly match the performance of the final product, as there'll likely be some additional checks (especially when there's a __future__ directive to concern ourselves with), but it's a start. yield from: https://github.com/Rosuav/GenStopIter/blob/485d1/perftest.py explicit loop: https://github.com/Rosuav/GenStopIter/blob/c071d/perftest.py The numbers are pretty noisy, but I'm seeing about a 5% slowdown in the 'yield from' version, with a recursion depth of 100 generators. (Obviously less percentage slowdown with less depth, as other factors have more impact.) Rewriting the loop to use an explicit try/except and while loop roughly doubles the time taken (so clearly 'yield from' is implemented very efficiently), and also destroys any meaning in the numbers - the two interpreters come out effectively equal. My preliminary conclusion is that there is some impact, but it's unlikely to be significant in the real world. Do you have a more real-world code example to try? > After thinking about that concern for a while, I'd like to suggest the > idea of having a new builtin "allow_implicit_stop" decorator that > swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" > flag set for one with it cleared (attempting to apply > "allow_implicit_stop" to a normal function would be an error). > > Then the updated version of the above example would become: > > @allow_implicit_stop > def my_generator(): > ... > yield next(it) > ... > > Which would be semantically equivalent to: > > def my_generator(): >try: >... > yield next(it) > ... > except StopIteration > return > > but *much* faster (especially if used in a producer/consumer pipeline) > since it would allow a single StopIteration instance to propagate > through the entire pipeline, rather than creating and destroying new > ones at each stage. If the issue is performance, I would prefer to see something done with a peephole optimizer instead: if there's a "try... except StopIteration: return" construct, it's optimized away down to a magic flag. That way, the code is guaranteed to be correct in all cases, with no hidden behavioral changes, and this is _just_ a performance optimization. I'd still rather see the exception-catching scope narrowed as much as possible, though, which means not having something that's semantically equivalent to wrapping the whole generator in "try... except StopIteration: pass". > P.S. While I'm less convinced this part is a good idea, if > "allow_implicit_stop" accepted both generator functions *and* > generator objects, then folks could even still explicitly opt in to > the "or stop()" trick - and anyone reading the code would have a name > to look up to see what was going on. That would look somewhat thus, then: use_an_iterable(allow_implicit_stop(x+1 for x in itr if x<10 or stop())) I'm definitely not convinced that this would improve matters. However, I don't currently have any recommendation for an "or stop()" replacement, other than "refactor it into an explicitly-written generator function and call it, then you can use return statements". Suggestions welcomed. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 2:55 AM, Hrvoje Niksic wrote: > To retrieve a single value from an iterator, one can use the for/break/else > idiom: > > def my_generator(): > ... > for val in it: > yield val > break > else: > return > ... While that does work, it's not really much more "normal" than a try/except. A for loop implies iteration; having a loop with a hard "break" at the end of it messes with expectations. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 9:53 AM, Nick Coghlan wrote: > The implicit stop decorator would then check the flags on the code object > attached to the passed in function. If GENERATOR wasn't set, that would be > an immediate ValueError, while if EXPLICIT_STOP wasn't set, the generator > function would be passed through unmodified. However, if EXPLICIT_STOP *was* > set, the generator function would be replaced by a *new* generator function > with a *new* code object, where the only change was to clear the > EXPLICIT_STOP flag. Is it possible to replace the code object without replacing the function? Imagine if you have multiple decorators, one of which retains a reference to the function and then this one which replaces it - the order of decoration would be critical. OTOH, I don't know that anyone would retain references to __code__. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 11:33 AM, Guido van Rossum wrote: > The design just copies the code object with one flag set differently. Code > objects are immutable but they can be copied (though the interface to do > that is kind of hidden). Yes, but the proposal as written spoke of replacing the generator *function*, which has broader consequences. If it's simply replacing the __code__ attribute of that function, it ought to be safe, I think? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 11:50 AM, Guido van Rossum wrote: > No, that was a figure of speech. The proposed decorator returns a new > function object that references a new code object. The original function and > code object are unchanged. Then it has a potentially-confusing interaction with decorators like Flask's app.route(), which return the original function unchanged, but also save a reference to it elsewhere. The order of decoration determines the effect of the @hettinger decorator; there will be two functions around which are almost, but not entirely, identical, and it'd be very easy to not notice that you decorated in the wrong order. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 12:01 PM, Guido van Rossum wrote: > Well, that's just a general problem with decorator ordering. Indeed. I was hoping it could be avoided in this instance by just altering __code__ on an existing function, but if that's not possible, we fall back to what is, after all, a known and documented concern. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 12:11 PM, Guido van Rossum wrote: > A decorator with a side effect *elsewhere* (like the route registrations) is > acceptable; one with a side effect *on the decorated function* is > questionable, and instead the decorator should behave "functionally", i.e. > return a new object instead. Okay! I learn something new every day. :) Did not know that... so it's a good thing I don't write decorators. I withdraw the suggestion. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 27, 2014 at 8:57 AM, Isaac Schwabacher wrote: >> Can you summarize that in a self-contained form for inclusion in the PEP? >> >> (That was a rhetorical question. :-) > > Sure. Is it on GitHub? ;D Thanks Isaac, I've incorporated your edits. https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479 and asyncio
On Fri, Nov 28, 2014 at 8:54 AM, Victor Stinner wrote: > def return_value(value): > if 0: > yield > raise Return(value) This is one known significant backward-incompatibility issue with this PEP - it'll be difficult to make this work on Python 2.7, where "return value" would be a problem, and 3.7, where "raise StopIteration" would be a problem. At present, I don't know of a solution to this. In 3.x-only code, you could simply use 'return value' directly; in 2.7 code, StopIteration doesn't seem to even *have* a .value attribute (and .args[0] has to be used instead). But I don't like the idea of a "from __past__" directive. It means backward-compatibility code has to be maintained through eternity (otherwise it just shifts the problem to "why did you remove my __past__ directive, I want a from __paster__ import division"), which means both the Python implementation code (every Python, not just CPython) needs to cope, *and* everyone who reads Python code needs to cope. For python-list, Stack Overflow, and other such coding help places, this means more questions to ask about a piece of code. For real-world usage, it means scanning back up to the top of the file every time you read something that's been affected by a __past__ directive. Plus, which __future__ directives need __past__ equivalents? Personally, I wouldn't bother making "from __past__ import lack_of_with_statement", but your friend is wanting "division", and I'm sure "print_statement" would be wanted... and, this is the one that'd split everyone and put the sides to war: "bytes_literals". Personally, I would want python-dev to say "There will NEVER be a from __past__ import bytes_literals directive", but there are going to be others who say "But my code would be so much cleaner AND faster if you do!", and IMO this is a good reason to avoid having any __past__ directives at all. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479 and asyncio
On Fri, Nov 28, 2014 at 8:18 PM, Victor Stinner wrote: > 2014-11-28 10:12 GMT+01:00 Greg Ewing : >> I don't understand. If I'm interpreting PEP 479 correctly, in >> 'x = yield from foo', a StopIteration raised by foo.__next__() >> doesn't get turned into a RuntimeError > > The Trollius coroutine uses "raise Return(value)" which is basically a > "raise StopIteraton(value)", and this is forbidden by the PEP 479. > With the PEP 479, the StopIteration is replaced with a RuntimeError. The question, I guess, is: Why can't it be translated into "return value"? One answer is: Because that's not legal in Python 2.7. And I can't respond to that answer, unfortunately. That's the one major backward compat issue. (Another answer may be "Because it would require changes to many intermediate generators, not all of which are under our control". If that's the issue, then it'll simply be a matter of telling people "When you upgrade to Python 3.6, you will start to see warnings unless you make this change".) ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] advice needed: best approach to enabling "metamodules"?
On Sat, Nov 29, 2014 at 12:59 PM, Nathaniel Smith wrote: > Option 4: Add a new function sys.swap_module_internals, which takes > two module objects and swaps their __dict__ and other attributes. By > making the operation a swap instead of an assignment, we avoid the > lifecycle pitfalls from Option 3. By making it a builtin, we can make > sure it always handles all the module fields that matter, not just > __dict__. Usage: > >new_module = MyModuleSubclass(...) >sys.swap_module_internals(new_module, sys.modules[__name__]) >sys.modules[__name__] = new_module > > Option 4 downside: Obviously a hack. This one corresponds to what I've seen in quite a number of C APIs. It's not ideal, but nothing is; and at least this way, it's clear that you're fiddling with internals. Letting the interpreter do the grunt-work for you is *definitely* preferable to having recipes out there saying "swap in a new __dict__, then don't forget to clear the old module's __dict__", which will have massive versioning issues as soon as a new best-practice comes along; making it a function, like this, means its implementation can smoothly change between versions (even in a bug-fix release). Would it be better to make that function also switch out the entry in sys.modules? That way, it's 100% dedicated to this job of "I want to make a subclass of module and use that for myself", and could then be made atomic against other imports. I've no idea whether there's any other weird shenanigans that could be deployed with this kind of module switch, nor whether cutting them out would be a good or bad thing! ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On Sun, Nov 30, 2014 at 11:37 AM, Donald Stufft wrote: > I also don’t know how to do this. When I’m doing multiple things for CPython > my “branching” strategy is essentially using hg diff to create a patch file > with my “branch” name (``hg diff > my-branch.patch``), then revert all of my > changes (``hg revert —all —no-backup``), then either work on a new “branch” > or switch to an old “branch” by applying the corresponding patch > (``patch -p1 < other-branch.patch``). IMO, this is missing out on part of the benefit of a DVCS. When your patches are always done purely on the basis of files, and have to be managed separately, everything will be manual; and your edits won't (normally) contain commit messages, authorship headers, date/time stamps, and all the other things that a commit will normally have. Using GitHub automatically makes all that available; when someone forks the project and adds a commit, that commit will exist and have its full identity, metadata, etc, and if/when it gets merged into trunk, all that will be carried through automatically. I strongly support this PEP. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479
On Sun, Nov 30, 2014 at 1:04 PM, Jim J. Jewett wrote: > I have a strong suspicion that I'm missing something; I have been > persuaded both directions too often to believe I have a grip on the > real issue. > > So I'm putting out some assumptions; please tell me if I'm wrong, and > maybe make them more explicit in the PEP. > > (1) The change will only affect situations where StopIteration is > currently raised as an Exception -- i.e., it leaks past the bounds of > a loop. Where a StopIteration would come up out of the generator. Inside the generator function, it's exactly the same as it is in any other function; you can raise it, you can catch it, everything's normal. > (2) This can happen because of an explicit raise StopIteration. This > is currently a supported idiom, and that is changing with PEP 479. Correct. There is nothing that explicitly-raised StopIteration can do in 3.0-3.4 that a return statement can't do in 3.0-3.7. There is the downside that "raise StopIteration(value)" works on 2.7 where "return value" is a syntax error; the PEP currently has no solution for this. > (2a) Generators in the unwind path will now need to catch and reraise. More likely, catch and return; if your code was allowing "next(iter)" to have the effect of potentially terminating the function, then you now have to spell that "try: next(iter); except StopIteration: return", which makes it clear that there's control flow here. > (3) It can also happen because of an explicit next statement (as > opposed the the implicit next of a loop). > This is currently supported; after PEP 479, the next statement should > be wrapped in a try statement, so that the intent will be explicit. Correct, as per previous point. As you say, the intent will be explicit: take a value, and if there aren't any more, stop processing. > (4) It can happen because of "yield from" yielding from an iterator, > rather than a generator? No; as I understand it (though maybe I'm wrong too), "yield from" will yield every value the other iterator yields, and will then quietly emit a value if the iterator raises StopIteration, or will allow any other exception to propagate. The StopIteration coming from the iterator is absorbed by the "yield from" construct. To completely propagate it out, "return (yield from iter)" should cover all three results (yielded value, returned value, raised exception). > (5) There is no other case where this can happen? (So the generator > comprehension case won't matter unless it also includes one of the > earlier cases.) Correct. In a generator expression (I assume that's what you mean?), the most likely way to leak a StopIteration is the "or stop()" hack, which has always been at least slightly dubious, and is now going to be actively rejected. Control flow in a generator expression is now the same as in a comprehension, with no early-abort option; if you want that, the best way is to break the expression into an out-of-line generator function. This is now very similar to the restrictions on lambda; you can't (eg) raise exceptions in a lambda function, and if anyone comes to python-list asking how to do so, the best response is "use def instead of lambda". ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On Sun, Nov 30, 2014 at 4:06 PM, Ben Finney wrote: > I don't get a vote. So I'm glad there are some within the Python core > development team that can see the mistakes inherent in depending on > non-free tools for developing free software. While this is a laudable view, this kind of extreme stance is contrary to any semblance of practicality. Compare: http://www.gnu.org/distros/free-distros.html http://www.gnu.org/distros/common-distros.html#Debian Debian is not considered sufficiently free because "people can readily learn about these nonfree packages by browsing Debian's online package database", even though you have to be very much explicit about these things (you have to go and enable the non-free repos). Yes, GitHub is proprietary. But all of your actual code is stored in git, which is free, and it's easy to push that to a new host somewhere else, or create your own host. This proposal is for repositories that don't need much in the way of issue trackers etc, so shifting away from GitHub shouldn't demand anything beyond moving the repos themselves. How bad is it, really? Is it worth fighting a philosophical battle for the sake of no real gain, sacrificing real benefits for the intangible "but it's not free" debate? Python is already using quite a bit of non-free software in its ecosystem. The Windows builds of CPython are made with Microsoft's compiler, and the recent discussion about shifting to Cygwin or MinGW basically boiled down to "but it ought to be free software", and that was considered not a sufficiently strong argument. In each case, the decision has impact on other people (using MSVC for the official python.org installers means extension writers need to use MSVC too; and using GitHub means that contributors are strongly encouraged, possibly required, to use GitHub); so why is it acceptable to use a non-free compiler, but not acceptable to use a non-free host? I admire and respect the people who, for their own personal use, absolutely and utterly refuse to use any non-free systems or software. It's great that they do it, because that helps encourage free software to be created. But for myself? I'll use whatever makes the most sense. Proprietary systems have inherent issues (the best-maintained non-free programs seem to have about the same bugginess as a poorly-maintained free program, or at least that's how it feels), but if the available free alternatives have even more issues, I'll not hobble myself for the purity of freedom. Practicality wins. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On Sun, Nov 30, 2014 at 8:54 PM, Nick Coghlan wrote: > On 30 November 2014 at 15:23, Chris Angelico wrote: >> Python is already using quite a bit of non-free software in its >> ecosystem. The Windows builds of CPython are made with Microsoft's >> compiler, and the recent discussion about shifting to Cygwin or MinGW >> basically boiled down to "but it ought to be free software", and that >> was considered not a sufficiently strong argument. In each case, the >> decision has impact on other people (using MSVC for the official >> python.org installers means extension writers need to use MSVC too; >> and using GitHub means that contributors are strongly encouraged, >> possibly required, to use GitHub); so why is it acceptable to use a >> non-free compiler, but not acceptable to use a non-free host? > > Relying on non-free software to support users of a non-free platform > is rather different from *requiring* the use of non-free software to > participate in core Python community design processes. But what non-free software is required to use the community design processes? The GitHub client is entirely optional; I don't use it, I just use git itself. Using a free client to access a proprietary server isn't the same as using non-free software. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github
On Mon, Dec 1, 2014 at 6:28 AM, Ethan Furman wrote: > My issues with GitHub range from selfish to philosophical: > > - (selfish) I don't want to learn git This ties in directly with the popularity argument. How many people are there who know hg and don't know git? How many who know git and don't know hg? So while this is a git issue for you, it's going to be a Mercurial issue for a lot more people. (And even people like me who kinda know both are going to spend a lot more time with git than with hg, on average. My knowledge of git is fairly advanced - I use git hooks, have scripts that manage things for me, can repair a repository that's seen some mess in it, etc - but my Mercurial knowledge is basic - I can clone and pull, not even sure I can push as I have literally never done so, and I basically turn to a dev guide to figure out how to make a patch in the appropriate way.) > - (practical) from what I hear git can have issues with losing history -- > in a > project that has many volunteer and part-time developers, using a tool > that > can munge your data just doesn't seem very wise It is possible to rewrite history in git, but - assuming you're not going to go to the ridiculous extent of SHA-1 cracking - it'll always be done by creating a new stream of commits, and git is very careful about confirming that you want to do this sort of thing. When you have a central server, you can simply configure it to reject any non-fast-forward pushes. (I'm not sure how to set receive.denyNonFastForwards on GitHub, but it's likely to be possible. In any case, you can always have an authoritative clone on python.org somewhere which mandates this kind of thing.) The git philosophy is: Your repository is yours. What you do with it is up to you. If that means rewriting history, git will confirm with you that you really want to do that, and then go right ahead and do what you ask. But then if your repo is a clone of someone else's, well, that someone else controls the other repo, so you might well not be allowed to merge changes in. > - (practical) supporting git and hg means learning two different workflows Already a problem for a lot of people. Unless CPython is the only project you ever contribute to, chances are you're going to meet git somewhere. If CPython used some mindblowingly brilliant but utterly unheard-of DVCS, sure it might be possible for core devs to avoid using any of the popular systems, but nobody else could. Personally, and somewhat selfishly, I would love to see the PEPs repo move to GitHub. For the two PEPs that I wrote, I had to juggle my own personal PEP draft repo and the central Mercurial peps repo; making sure changes got deployed from one to the other was basically a manual job, without any tool support. If I could send PRs against a clone of the peps repo, I would work that way instead of creating a dedicated repo at all (at least until such time as I need other files; the PEP 463 repo grew a few other bits and bobs, but they could just as easily have been in an ancillary repo without the PEP text in it). There'll be easily as many people who selfishly want git as selfishly want hg :) ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unicode decode exception
On Sun, Nov 30, 2014 at 7:07 PM, balaji marisetti
wrote:
> Hi,
Hi. This list is for the development *of* Python, not development
*with* Python, so I'm sending this reply also to
python-l...@python.org where it can be better handled. You'll probably
want to subscribe here:
https://mail.python.org/mailman/listinfo/python-list
or alternatively, point a news reader at comp.lang.python. Let's
continue this conversation on python-list rather than python-dev.
> When I try to iterate through the lines of a
> file("openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c"), I get a
> UnicodeDecodeError (in python 3.4.0 on Ubuntu 14.04). But there is no
> such error with python 2.7.6. What could be the problem?
The difference between the two Python versions is that 2.7 lets you be
a bit sloppy about Unicode vs bytes, but 3.4 requires that you keep
them properly separate.
> In [39]: with open("openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c") as f:
> for line in f:
> print (line)
>
> ---
> UnicodeDecodeErrorTraceback (most recent call last)
> in ()
> 1 with open("../openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c") as f:
> > 2 for line in f:
> 3 print (line)
> 4
>
> /usr/lib/python3.4/codecs.py in decode(self, input, final)
> 311 # decode input (taking the buffer into account)
> 312 data = self.buffer + input
> --> 313 (result, consumed) = self._buffer_decode(data,
> self.errors, final)
> 314 # keep undecoded input until the next call
> 315 self.buffer = data[consumed:]
>
>
> --
> :-)balaji
Most likely, the line of input that you just reached has a non-ASCII
character, and the default encoding is ASCII. (Though without the
actual exception message, I can't be sure of that.) The best fix would
be to know what the file's encoding is, and simply add that as a
parameter to your open() call - perhaps this:
with open("filename", encoding="utf-8") as f:
If you use the right encoding, and the file is correctly encoded, you
should have no errors. If you still have errors... welcome to data
problems, life can be hard. :|
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Joining the PEP Editors team
In response to Guido's call for volunteers, I'm offering myself as a PEP editor. Who is in charge of this kind of thing? Who manages public key lists etc? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition
On Sat, Dec 13, 2014 at 4:29 PM, Donald Stufft wrote: > So that's basically it, lowest common demoniator programming where it's hard > to > look at the future and see anything but the same (or similar) language subset > that I'm currently using. This is especially frustrating when you see other > languages doing cool and interesting new things and it feels like we're stuck > with what we had in 2008 or 2010. That's what happens when you want to support a version of Python that was released in 2008 or 2010. Perhaps the impetus for people to move onto Python 3 has to come from people like you saying "I'm not going to support 2.7 any more as of version X.Y", and letting them run two interpreters. It's really not that hard to keep 2.7 around for what expects it, and 3.4/3.5/whatever for everything else. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition
On Sat, Dec 13, 2014 at 5:13 PM, Donald Stufft wrote: > First of all, it's essentially the route that Python itself took and the side > effects of that is essentially what is making things less-fun for me to write > Python. Doing the same to the users of the things I write would make me feel > bad that I was forcing them to either do all the work to port their stuff > (and any dependencies) just so they can use a newer version of my library. Ultimately, those programs WILL have to be migrated, or they will have to remain on an unsupported system. You have the choice of either continuing to do what you find un-fun (cross-compatibility code) until 2020 and maybe beyond, or stopping support for 2.7 sooner than that. All you're doing is changing *when* the inevitable migration happens. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Fwd: fixing broken link in pep 3
On Fri, Dec 19, 2014 at 5:39 AM, Guido van Rossum wrote: >-- Forwarded message -- > From: Victor Stinner > > Hi, > > Yes, the link is dead. It looks like the following link contains the same > info: > https://docs.python.org/devguide/triaging.html > > Dead page: > https://web.archive.org/web/20090704040931/http://www.python.org/dev/workflow/ > "Core Development > Issue Workflow" > > Victor Edits made to PEP 3, link now updated. Noticed along the way that the next link down (for people _submitting_ bugs) is pointing to the /2/ section of the docs; should that be updated to send people to /3/, or are the two kept in sync? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Fwd: fixing broken link in pep 3
On Fri, Dec 19, 2014 at 12:24 PM, Terry Reedy wrote: > PEP 3 is listed in PEP 0 under Abandoned, Withdrawn, and Rejected PEPs > If this is proper, it does not make sense to update it. > If this is not, the header should be updated. Guido passed the request on to the pep-editors list, which I took to mean that this should be updated. PEP 3 has been replaced with info in the dev guide, and the link in question is to the exact page of that dev guide which replaces it. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Email from Rietveld Code Review Tool is classified as spam
On Thu, Dec 25, 2014 at 2:56 PM, Sky Kok wrote: > Anyway, sometimes when people review my patches for CPython, they send > me a notice through Rietveld Code Review Tool which later will send an > email to me. However, my GMail spam filter is aggressive so the email > will always be classified as spam because it fails spf checking. So if > Taylor Swift clicks 'send email' in Rietveld after reviewing my patch, > Rietveld will send email to me but the email pretends as if it is sent > from tay...@swift.com. Hence, failing spf checking. > > Maybe we shouldn't pretend as someone else when sending email through > Rietveld? That's not the fault of Gmail, except perhaps in that no rejection will have gone to the originating server. The solution is exactly as you say. The "From" can still say tay...@swift.com, but the envelope-from (the "MAIL FROM:" at protocol level) should be an address that can cope with bounces. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Google search labels Python 2.7 docs as Python 3.4
On Thu, Jan 1, 2015 at 12:24 PM, Benjamin Peterson wrote: > On Wed, Dec 31, 2014, at 19:32, Ryan Gonzalez wrote: >> Not sure if this is something to post here...but... >> >> [image: Inline image 1] > > That must be some sort of inconsistency in Google's index, since the > actual page's title is correct. This issue has persisted for a while. I first noticed it in early December (though it may have been there longer), and ignored it on the assumption that it was a transitional state. I don't know how long a transitional state can last, though. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Windows installer for Python 3.5
On Sun, Jan 4, 2015 at 10:34 AM, Steve Dower wrote: > http://stevedower.id.au/blog/the-python-3-5-installer/ You talk of installing by default into Program Files, and having a separate per-user installation mode. How do these two installation targets interact? Suppose someone installs 3.5 globally, then installs 3.6 for self only? Or installs 3.5.1 for self only? I would normally expect a per-user installation to trump a global one, but this could make for a lovely dep-hell on a system that's used by one person, who then isn't sure what was installed as admin and what wasn't. (Also: I'm assuming that pip would require admin privs if Python is in Program Files, and won't require admins if it's per-user, right? Same as the Python installer itself?) ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Windows installer for Python 3.5
On Mon, Jan 5, 2015 at 9:56 AM, Steve Dower wrote:
> (Windows will always put per-user PATH entries at the end). The py.exe
> launcher handles this best, and the system version will always be found first.
>
> As for versions, 3.5 vs 3.6 should be distinguished explicitly ("py -3.5" vs
> "py -3.6") and 3.5.0 vs 3.5.1 can only be installed at the same time if one
> is system-wide and the other is per-user. In this case, the per-user one will
> be used by py.exe, even if it is older than the system-wide one.
>
Wait, what?
If I'm reading this correctly, PATH is set such that an all-users
version trumps a self-only version, but when you use the py.exe
launcher, it's the other way around? That seems extremely confusing.
But if that's not a deliberate design decision and isn't enforced by
external code, would it be possible to simply invert that for PATH,
and have a per-user installation always be found ahead of a
system-wide one? That would make reasonable sense, if it can be done.
> Incidentally, whoever came up with the py.exe launcher deserves a medal. It's
> made dealing with multiple versions amazingly easy.
https://www.python.org/dev/peps/pep-0397/ - and I agree :)
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Windows installer for Python 3.5
On Mon, Jan 5, 2015 at 12:20 PM, Steve Dower wrote: > Unfortunately, Windows enforces the PATH ordering. It constructs the PATH > from two registry keys, one is the system-wide value (that requires > administrative privileges to modify) and it is followed by the user's value > (that does not require administrative privileges). This is probably for > security reasons and can't be changed. > > PATH also suffers from including the most-recently installed Python version > first, rather than the most recent version. Basically, py.exe gives the > behaviour we want and PATH simply can't do it. > > I think this means the best way to make multiple versions work properly is to > rename python.exe to python3.5.exe, then install the launcher as python.exe > and python3.exe (with some logic to look at its own name) so it can resolve > the right version. Maybe we can even extend the launcher to resolve launchers > in Scripts (pip.exe, etc.) and have consistent rules there too? > > Or perhaps this is more trouble than it's worth, and we should let "py.exe" > have a fixed set of rules and let PATH work as people have come to expect. > Thoughts? Great. :( Well, if there's no changing the PATH behaviour, so be it... the question is, should py.exe match that, or do the sane thing? Neither option is ideal. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
