Re: [Python-Dev] New PEP
On 03/21/2012 07:39 PM, Huan Do wrote: *Hi, I am a graduating Berkeley student that loves python and would like to propose an enhancement to python. My proposal introduces a concept of slicing generator. For instance, if one does x[:] it returns a list which is a copy of x. Sometimes programmers would want to iterate over a slice of x, but they do not like the overhead of constructing another list. Instead we can create a similar operator that returns a generator. My proposed syntax is x(:). The programmers are of course able to set lower, upper, and step size like the following. x(1::-1) This would make code much cleaner in a lot of instances, one example lets say we have a very large list x and we want to sum all the numbers but the last 20, and we only want to loop through the even indices. We would have to do something like this. sum(x[:-20:2]) or we can do a workaround to save space for time and do something like this. sum( value for i, value in enumerate(x) if i < -20 and not i % 2 ) But with my proposal we are able do the following. sum(x(:-20:2)) Which affords space without sacrificing expressiveness. For another example lets say we have a problem that we want to check a condition is true for every pairwise element in a list x. def allfriends(x): for i in range(len(x)): for j in range(i+1, len(x)): if not friends(x[i], x[j]): return False return True A more pythonic way is to actually loop through the values instead of the indices like this. def allfriends(x): for i, a in enumerate(x): for j, b in enumerate(x[i+1:]): if not friends(a, b): return False return True This however bring a lot of overhead because we have to construct a new list for every slice call. With my proposal we are able to do this. def allfriends(x): for i, a in enumerate(x): for j, b in enumerate(x(i+1:)): if not friends(a, b): return False return True This proposal however goes against one heuristic in the zen of python, namely “Special cases aren’t special enough to break the rules.” The way that the proposal breaks this rule is because the syntax x(:), uses a function call syntax but would be a special case here. I chose using parenthesis because I wanted this operation to be analogous to the generator syntax in list comprehensions. ListGenerators Comprehension [ x for x in L ]( x for x in L ) Slicing L[a:b:c]L(a:b:c) Tell me what you guys think. Thanks!* ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/animelovin%40gmail.com Hi, I'm not sure i get it.. Assuming your PEP is accepted, what should happens then to the lambda op and standard function calls ? Or Is this merely another case of metaprogramming, which obviously should not be confused with languages such as lisp? Thank you, Etienne ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On 03/29/2012 04:48 PM, R. David Murray wrote: On Thu, 29 Mar 2012 16:31:03 -0400, "R. David Murray" wrote: On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: My original assessment was that this only affects dicts whose keys have a user-implemented __hash__ or __eq__ implementation, and that the number of apps that use this *and* assume the threadsafe property would be pretty small. This is just intuition, I don't have hard facts. But I do want to stress that not all dict lookups automatically become thread-unsafe, only those that need to run user code as part of the key lookup. You are probably correct, but the thing is that one still has to do the code audit to be sure...and then make sure that no one later introduces such an object type as a dict key. I just did a quick grep on our project. We are only defining __eq__ and __hash__ a couple places, but both are objects that could easily get used as dict keys (there is a good chance that's *why* those methods are defined) accessed by more than one thread. I haven't done the audit to find out :) The libraries we depend on have many more definitions of __eq__ and __hash__, and we'd have to check them too. (Including SQLAlchemy, and I wouldn't want that job.) So our intuition that this is not common may be wrong. --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/animelovin%40gmail.com Hm, as far I understand this seems like an issue for gnu PTH, not python job, which should transparently handles thread safety issues based on the host/machine capabilities. Therefore I hope the fix in python don't affect thread-unsafe apps to raise spurious RuntimeErrors when a dict get modified across a SMP-aware platform... :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On 03/29/2012 06:07 PM, R. David Murray wrote: On Thu, 29 Mar 2012 23:00:20 +0200, Stefan Behnel wrote: R. David Murray, 29.03.2012 22:31: On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: Some of us have expressed uneasiness about the consequences of dict raising an error on lookup if the dict has been modified, the fix Victor made to solve one of the crashers. I don't know if I speak for the others, but (assuming that I understand the change correctly) my concern is that there is probably a significant amount of threading code out there that assumes that dict *lookup* is a thread-safe operation. Much of that code will, if moved to Python 3.3, now be subject to random runtime errors for which it will not be prepared. Further, code which appears safe can suddenly become unsafe if a refactoring of the code causes an object to be stored in the dictionary that has a Python equality method. My original assessment was that this only affects dicts whose keys have a user-implemented __hash__ or __eq__ implementation, and that the number of apps that use this *and* assume the threadsafe property would be pretty small. This is just intuition, I don't have hard facts. But I do want to stress that not all dict lookups automatically become thread-unsafe, only those that need to run user code as part of the key lookup. You are probably correct, but the thing is that one still has to do the code audit to be sure...and then make sure that no one later introduces such an object type as a dict key. The thing is: the assumption that arbitrary dict lookups are GIL-atomic has *always* been false. Only those that do not involve Python code execution for the hash key calculation or the object comparison are. That includes the built-in strings and numbers (and tuples of them), which are by far the most common dict keys. Looking up arbitrary user provided objects is definitely not guaranteed to be atomic. Well, I'm afraid I was using the term 'thread safety' rather too loosely there. What I mean is that if you do a dict lookup, the lookup either returns a value or a KeyError, and that if you get back an object that object has internally consistent state. The problem this fix introduces is that the lookup may fail with a RuntimeError rather than a KeyError, which it has never done before. I think that is what Guido means by code that uses objects with python eq/hash *and* assumes threadsafe lookup. If mutation of the objects or dict during the lookup is a concern, then the code would use locks and wouldn't have the problem. But there are certainly situations where it doesn't matter if the dictionary mutates during the lookup, as long as you get either an object or a KeyError, and thus no locks are (currently) needed. Maybe I'm being paranoid about breakage here, but as with most backward compatibility concerns, there are probably more bits of code that will be affected than our intuition indicates. --David ___ what this suppose to mean exactly? To "mutate" is a bit odd concept for a programming language I suppose. Also I suppose I must be missing something which makes you feel like this is an OT post when the problem seem most likely to be exclusively in python 3.3, another reason I guess to not upgrade yet all that massively using 2to3. :-) cheers, Etienne ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
Hi Guido, I'm sorry for being unclear! I just try actually to learn what thoses consequences for theses 'unattended' mutations in dictionary key lookups could be, :-) however, it seems now that I might have touch a nerve without realizing it. I would therefore appreciate more light on this "issue" if you like to enlighten us all. :D Regards, Etienne On 03/30/2012 10:47 AM, Guido van Rossum wrote: Etienne, I have not understood either of your messages in this thread. They just did not make sense to me. Do you actually understand the issue at hand? --Guido On Friday, March 30, 2012, Etienne Robillard wrote: On 03/29/2012 06:07 PM, R. David Murray wrote: On Thu, 29 Mar 2012 23:00:20 +0200, Stefan Behnel wrote: R. David Murray, 29.03.2012 22:31: On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: Some of us have expressed uneasiness about the consequences of dict raising an error on lookup if the dict has been modified, the fix Victor made to solve one of the crashers. I don't know if I speak for the others, but (assuming that I understand the change correctly) my concern is that there is probably a significant amount of threading code out there that assumes that dict *lookup* is a thread-safe operation. Much of that code will, if moved to Python 3.3, now be subject to random runtime errors for which it will not be prepared. Further, code which appears safe can suddenly become unsafe if a refactoring of the code causes an object to be stored in the dictionary that has a Python equality method. My original assessment was that this only affects dicts whose keys have a user-implemented __hash__ or __eq__ implementation, and that the number of apps that use this *and* assume the threadsafe property would be pretty small. This is just intuition, I don't have hard facts. But I do want to stress that not all dict lookups automatically become thread-unsafe, only those that need to run user code as part of the key lookup. You are probably correct, but the thing is that one still has to do the code audit to be sure...and then make sure that no one later introduces such an object type as a dict key. The thing is: the assumption that arbitrary dict lookups are GIL-atomic has *always* been false. Only those that do not involve Python code execution for the hash key calculation or the object comparison are. That includes the built-in strings and numbers (and tuples of them), which are by far the most common dict keys. Looking up arbitrary user provided objects is definitely not guaranteed to be atomic. Well, I'm afraid I was using the term 'thread safety' rather too loosely there. What I mean is that if you do a dict lookup, the lookup either returns a value or a KeyError, and that if you get back an object that object has internally consistent state. The problem this fix introduces is that the lookup may fail with a RuntimeError rather than a KeyError, which it has never done before. I think that is what Guido means by code that uses objects with python eq/hash *and* assumes threadsafe lookup. If mutation of the objects or dict during the lookup is a concern, then the code would use locks and wouldn't have the problem. But there are certainly situations where it doesn't matter if the dictionary mutates during the lookup, as long as you get either an object or a KeyError, and thus no locks are (currently) needed. Maybe I'm being paranoid about breakage here, but as with most backward compatibility concerns, there are probably more bits of code that will be affected than our intuition indicates. --David __
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
"Multiple threads can agree by convention not to mutate a shared dict, there's no great need for enforcement. Multiple processes can't share dicts." its not sure I get completely the meaning of "mutate"... And if possible, I would like also the rational for the 2nd phrase while we're at it as it seem a little unclear too. Sorry also if this is OT... :) Regards, Etienne http://www.python.org/dev/peps/pep-0416/ On 03/30/2012 10:47 AM, Guido van Rossum wrote: Etienne, I have not understood either of your messages in this thread. They just did not make sense to me. Do you actually understand the issue at hand? --Guido On Friday, March 30, 2012, Etienne Robillard wrote: On 03/29/2012 06:07 PM, R. David Murray wrote: On Thu, 29 Mar 2012 23:00:20 +0200, Stefan Behnel wrote: R. David Murray, 29.03.2012 22:31: On Thu, 29 Mar 2012 13:09:17 -0700, Guido van Rossum wrote: On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray wrote: Some of us have expressed uneasiness about the consequences of dict raising an error on lookup if the dict has been modified, the fix Victor made to solve one of the crashers. I don't know if I speak for the others, but (assuming that I understand the change correctly) my concern is that there is probably a significant amount of threading code out there that assumes that dict *lookup* is a thread-safe operation. Much of that code will, if moved to Python 3.3, now be subject to random runtime errors for which it will not be prepared. Further, code which appears safe can suddenly become unsafe if a refactoring of the code causes an object to be stored in the dictionary that has a Python equality method. My original assessment was that this only affects dicts whose keys have a user-implemented __hash__ or __eq__ implementation, and that the number of apps that use this *and* assume the threadsafe property would be pretty small. This is just intuition, I don't have hard facts. But I do want to stress that not all dict lookups automatically become thread-unsafe, only those that need to run user code as part of the key lookup. You are probably correct, but the thing is that one still has to do the code audit to be sure...and then make sure that no one later introduces such an object type as a dict key. The thing is: the assumption that arbitrary dict lookups are GIL-atomic has *always* been false. Only those that do not involve Python code execution for the hash key calculation or the object comparison are. That includes the built-in strings and numbers (and tuples of them), which are by far the most common dict keys. Looking up arbitrary user provided objects is definitely not guaranteed to be atomic. Well, I'm afraid I was using the term 'thread safety' rather too loosely there. What I mean is that if you do a dict lookup, the lookup either returns a value or a KeyError, and that if you get back an object that object has internally consistent state. The problem this fix introduces is that the lookup may fail with a RuntimeError rather than a KeyError, which it has never done before. I think that is what Guido means by code that uses objects with python eq/hash *and* assumes threadsafe lookup. If mutation of the objects or dict during the lookup is a concern, then the code would use locks and wouldn't have the problem. But there are certainly situations where it doesn't matter if the dictionary mutates during the lookup, as long as you get either an object or a KeyError, and thus no locks are (currently) needed. Maybe I'm being paranoid about breakage here, but as with most backward compatibility concerns, there are probably more bits of code t
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
you wish...are you also truth allergic or irritated by the consequences of free speech ? Please stop giving me orders. You don't even know me and this is at all not necessary and good netiquette if you want to bring a point to ponder. Sorry for others who thinks this is not OT as I its probably related to pep-416 refusal. Cheers! Etienne On 03/30/2012 11:54 AM, Stefan Behnel wrote: Etienne Robillard, 30.03.2012 17:45: Sorry also if this is OT... :) Yes, it is. Please do as Nick told you. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
your reasoning is pathetic at best. i pass... Thanks for the tip :-) Cheers, Etienne On 03/30/2012 12:18 PM, Stefan Behnel wrote: Etienne Robillard, 30.03.2012 18:08: are you also truth allergic or irritated by the consequences of free speech ? Please note that "free speech" is a concept that is different from asking beginner's computer science questions on the core developer mailing list of a software development project. This is not the right forum to do so, and you should therefore move your "free speech" to one that is more appropriate. Nick has pointed you to one such forum and you would be well advised to use it - that's all I was trying to say. I hope it's clearer now. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/animelovin%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On 03/30/2012 02:23 PM, Ethan Furman wrote: Etienne Robillard wrote: your reasoning is pathetic at best. i pass... Thanks for the tip :-) The Python Developer list is for the discussion of developing Python, not for teaching basic programming. You are being rude, and a smiley does not make you less rude. I am adding you to my kill-file so I no longer see messages from you because you refuse to follow the advice you have been given. ~Ethan~ Add me to whatever file you want, but I believe the consequences for the new dict runtime errors just won't be resolved this way, and neither by systematically blocking alternative opinions as OT will help, because thats typically oppression, not free speech. :-) Cheers, :-) Etienne ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On 03/30/2012 03:02 PM, Guido van Rossum wrote: Hey Etienne, I am honestly trying to understand your contribution but you seem to have started a discussion about free speech. Trust me that we don't mind your contributions, we're just trying to understand what you're saying, and the free speech discussion isn't helping with that. I agree. So if you have a comment on the dict mutation problem, please say so. OK. If you need help understanding the problem, python-dev is not the place to ask questions; you could ask on the bug, or on the core-mentorship list as Nick suggested. But please stop bringing up free speech, that's not an issue. Guido, thanks for the wisdom and clarity of your reasoning. I really appreciate a positive attitude towards questioning not so obvious problems. So far I was only attempting to verify whether this is related to PEP-416 or not. If this is indeed related PEP 416, then I must obviously attest that I must still understand why a immutable dict would prevent this bug or not... Either ways, I fail to see where this is OT or should be discussed on a more obscur forum than python-dev. :-) Kind regards, Etienne ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On 03/30/2012 03:27 PM, Guido van Rossum wrote: On Fri, Mar 30, 2012 at 12:13 PM, Etienne Robillard wrote: On 03/30/2012 03:02 PM, Guido van Rossum wrote: Hey Etienne, I am honestly trying to understand your contribution but you seem to have started a discussion about free speech. Trust me that we don't mind your contributions, we're just trying to understand what you're saying, and the free speech discussion isn't helping with that. I agree. So if you have a comment on the dict mutation problem, please say so. OK. If you need help understanding the problem, python-dev is not the place to ask questions; you could ask on the bug, or on the core-mentorship list as Nick suggested. But please stop bringing up free speech, that's not an issue. Guido, thanks for the wisdom and clarity of your reasoning. I really appreciate a positive attitude towards questioning not so obvious problems. So far I was only attempting to verify whether this is related to PEP-416 or not. If this is indeed related PEP 416, then I must obviously attest that I must still understand why a immutable dict would prevent this bug or not... It's not related to PEP 416 (which was rejected). Please refer to http://bugs.python.org/issue14417 for the issue being discussed. Either ways, I fail to see where this is OT or should be discussed on a more obscur forum than python-dev. :-) We need to keep that list clear for important discussions. It is the only channel that the core Python developers have. If it has too much noise people will stop reading it and it stops functioning. Hence, we try to keep questions from newbies to a minimum -- there are other places where we welcome such questions though. So, once more, if you don't understand the issue and cannot figure it out by reading up, please ask somewhere else (or just accept that you don't have anything to contribute to this particular issue). This includes explaining basic terms like "mutate". On the other hand, if you *do* understand the problem, by all means let us know what you think of the question at hand (whether the change referred to in the issue is going to break people's code or not). We don't need more speculation though; that's how we got here in the first place (my speculation that it's not going to be an issue vs. RDM's speculation that it's going to cause widespread havoc :-). I hope you understand. No, not really. Anyways, I guess I'll have to further dig down why is PEP-416 is really important to Python and why it was likewise rejected, supposing I confused the pep 416 and issue 14417 along the way.. :-) CHeers, Etienne ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
The frozendict builtin type was rejected, but we are going to add types.MappingProxyType: see issue #14386. types.MappingProxyType(mydict.copy()) is very close to the frozendict builtin type. Victor Thanks, Victor. :) Will this mean the new dict subclass for CPython will not expose dictproxy to favorize a new types.MappingProxyType type to emulate immutable-like types ? What could be then consequences for code still expecting a non-mutable dict() type ? Therefore I guess this ticket provides more than just speculating points to reconsider such aliased types in cpython. I also found this article quite useful: http://www.cs.toronto.edu/~tijmen/programming/immutableDictionaries.html Yet I might miss how this "new dict" type could potentially induce a RuntimeError unless in python 3.3 a new dict proxy alias is introduced to perform invariant operations in non thread-safe code. Regards, Etienne ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 14417: consequences of new dict runtime error
On 03/30/2012 03:25 PM, R. David Murray wrote: On Fri, 30 Mar 2012 15:13:36 -0400, Etienne Robillard wrote: So far I was only attempting to verify whether this is related to PEP-416 or not. If this is indeed related PEP 416, then I must obviously attest that I must still understand why a immutable dict would prevent this bug or not... OK, that seems to be the source of your confusion, then. This has nothing to do with PEP-416. We are talking about issue Issue 14417 (like it says in the subject), which in turn is a reaction to the fix for issue 14205. --David Don't be so naive, David. This issue is more likely related to immutable dicts whether you like it or not, otherwise there would be no need to patch python 3.3 and include a new dict proxy type without exposing it fully. And secondly this is not only a speculation but my humble understanding of the interdependencies which seems to be related to the inclusion of a new dict proxy (immutable) mapper affecting invariably code expecting a mutable dictionary lookup to succeed whenever the dict structure has changed by (ie) overriding __hash__. Now if you don't mind, I don't mind to remind you how cell phones can be unsafe on an extended period of times for your health and brain so I really don't recommend their uses for cloud-based platforms requiring more advanced thread locking mecanism than what is being included in traditional CPython using standard (mutable) dicts. Regards, Etienne ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
