Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 06:22, Nick Coghlan wrote: > - download a futures module from PyPI and live with the additional > dependency Why would that be a problem? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010 08:11, Lennart Regebro wrote: > On Wed, May 26, 2010 at 06:22, Nick Coghlan wrote: >> - download a futures module from PyPI and live with the additional >> dependency > > Why would that be a problem? That has been hashed out repeatedly on this and other lists. Can it please be stipulated that for *some* people, in *some* cases, it is a problem? It seems to me that if you've experienced the sort of culture that makes it a problem, you understand the point immediately, but if you haven't, you never will (that's not disparaging anyone, the idiosyncracies of corporate culture are widespread and bizarre - if it helps, just remember that Dilbert is a documentary :-)) Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 09:54:13 +1000 Nick Coghlan wrote: > > > > What I would question here is what other things will be part > > of the "concurrent" package, and who will implement them. Are there > > plans for that? (or even tracker issues open?) > > I'm not sure it is called out explicitly in the PEP, but the specific > example that came up in the previous discussions was something like > "concurrent.pool" to hold a thread vs process agnostic worker pool > interface based on the existing Pool interface in multiprocessing > (with concrete implementations for both threading and > multiprocessing). Ha, I'm a bit surprised. Isn't it what "futures" already provides? (except that for some reason it insists on the "SomeExecutor" naming scheme) http://www.python.org/dev/peps/pep-3148/#processpoolexecutor Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 24, 2010, at 5:36 AM, Brian Quinlan wrote:
> On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote:
>> On May 23, 2010, at 2:37 AM, Brian Quinlan wrote:
>>> On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:
> ProcessPoolExecutor has the same serialization perils that multiprocessing
> does. My original plan was to link to the multiprocessing docs to explain
> them but I couldn't find them listed.
Linking to the pickle documentation might be a good start.
> Yes, the execution context is Executor-dependent. The section under
> ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think.
I suppose so. I guess I'm just looking for more precise usage of terminology.
(This is a PEP, after all. It's a specification that multiple VMs may have to
follow, not just some user documentation for a package, even if they'll
*probably* be using your code in all cases.) I'd be happier if there were a
clearer term than "calls" for the things being scheduled ("submissions"?),
since the done callbacks aren't called in the subprocess for
ProcessPoolExecutor, as we just discussed.
>> Sure. Really, almost any contract would work, it just needs to be spelled
>> out. It might be nice to know whether the thread invoking the callbacks is
>> a daemon thread or not, but I suppose it's not strictly necessary.
>
> Your concerns is that the thread will be killed when the interpreter exits?
> It won't be.
Good to know. Tell it to the PEP though, not me ;).
>> No reaction on [invoker vs. future]? I think you'll wish you did this in a
>> couple of years when you start bumping into application code that calls
>> "set_result" :).
>
> My reactions are mixed ;-)
Well, you are not obliged to take my advice, as long as I am not obliged to
refrain from mocking you mercilessly if it happens that I was right in a couple
of years ;-).
> Your proposal is to add a level of indirection to make it harder for people
> to call implementation methods. The downside is that it makes it a bit harder
> to write tests and Executors.
Both tests and executors will still create and invoke methods directly on one
object; the only additional difficulty seems to be the need to type '.future'
every so often on the executor/testing side of things, and that seems a cost
well worth paying to avoid confusion over who is allowed to call those methods
and when.
> I also can't see a big problem in letting people call set_result in client
> code though it is documented as being only for Executor implementations and
> tests.
>
> On the implementation side, I don't see why an Invoker needs a reference to
> the future.
Well, uh...
> class Invoker(object):
> def __init__(self):
> """Should only be called by Executor implementations."""
> self.future = Future()
^ this is what I'd call a "reference to the future"
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 09:37, Paul Moore wrote: > It seems to me that if you've experienced the sort of culture that > makes it a problem, Ah, it's a culture problem. In a heterogenous world, every action will benefit some and hurt some. Another arbitrary corporate ruleset could also mean you might be stuck on ancient python versions, and might not see a new module added to stdlib in 3.2 until 2015 or so. Some corporations go through a lot of trouble to prevent their employees from doing their job. Pythons core developers can not and should not let that hinder *them* from doing what is best for Python. Decisions on inclusion in stdlib must be made on what benefits Python and it's users in general. Since even small mistakes in a stdlib module will hurt far more people than having to having the module mature on PyPI until the worst API issues and bugs are ironed out, it's clear to me that letting a module mature on PyPI before inclusion is the better policy here, although how long obviously must be decided on a case by case basis. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 26, 2010, at 3:37 AM, Paul Moore wrote: > On 26 May 2010 08:11, Lennart Regebro wrote: >> On Wed, May 26, 2010 at 06:22, Nick Coghlan wrote: >>> - download a futures module from PyPI and live with the additional >>> dependency >> >> Why would that be a problem? > > That has been hashed out repeatedly on this and other lists. Can it > please be stipulated that for *some* people, in *some* cases, it is a > problem? Sure, but I for one fully support Lennart asking the question, because while in the short term this *is* a problem with packaging tools in the Python ecosystem, in the long term (as you do note) it's an organizational dysfunction that can be addressed with better tools. I think it would be bad to ever concede the point that sane factoring of dependencies and code re-use aren't worth it because some jerk in Accounting or System Operations wants you to fill out a requisition form for a software component that's free and liberally licensed anyway. To support the unfortunate reality that such jerks in such departments really do in fact exist, there should be simple tools to glom a set of small, nicely factored dependencies into a giant monolithic ball of crud that installs all at once, and slap a sticker on the side of it that says "I am only filling out your stupid form once, okay". This should be as distant as possible from the actual decision to package things in sensibly-sized chunks. In other words, while I kinda-sorta buy Brian's argument that having this module in easy reach will motivate more people to use a standard, tested idiom for parallelization, I *don't* think that the stdlib should be expanded simply to accommodate those who just don't want to install additional packages for anything.___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Nick Coghlan writes: > On 26/05/10 13:51, Stephen J. Turnbull wrote: > > People have been asking "what's special about this module, to violate > > the BCP principle?" There's nothing special about the fact that > > several people would use a "robust and debugged" futures module if it > > were in the stdlib. That's true of *every* module that is worth a > > PEP. > > The trick with futures and executor pools is that they're a *better* way > of programming with threads in many cases. and > However, given the choices of [...]. I'll choose the first option > every time, and my programs will be the worse for it. Again, nothing all that special about those; lots of proposed changes satisfy similar conditions. I don't think anyone denies the truth or applicability of those arguments. But are they enough? Really, what you're arguing is "now is better than never." Indeed, that is so. But you shouldn't forget that is immediately followed by "although never is often better than *right* now." ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010, at 18:44, Stephen J. Turnbull wrote: Nick Coghlan writes: On 26/05/10 13:51, Stephen J. Turnbull wrote: People have been asking "what's special about this module, to violate the BCP principle?" There's nothing special about the fact that several people would use a "robust and debugged" futures module if it were in the stdlib. That's true of *every* module that is worth a PEP. The trick with futures and executor pools is that they're a *better* way of programming with threads in many cases. and However, given the choices of [...]. I'll choose the first option every time, and my programs will be the worse for it. Again, nothing all that special about those; lots of proposed changes satisfy similar conditions. I don't think anyone denies the truth or applicability of those arguments. But are they enough? Really, what you're arguing is "now is better than never." Indeed, that is so. But you shouldn't forget that is immediately followed by "although never is often better than *right* now." I've been trying to stay out of the meta-discussions but "*right* now" would be >6 months if it applies in this context. If that is what "*right* now" means to you then I hope that I never have a heart attack in your presence and need an ambulance *right* now :-) Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010, at 18:09, Glyph Lefkowitz wrote:
On May 24, 2010, at 5:36 AM, Brian Quinlan wrote:
On May 24, 2010, at 5:16 AM, Glyph Lefkowitz wrote:
On May 23, 2010, at 2:37 AM, Brian Quinlan wrote:
On May 23, 2010, at 2:44 PM, Glyph Lefkowitz wrote:
ProcessPoolExecutor has the same serialization perils that
multiprocessing does. My original plan was to link to the
multiprocessing docs to explain them but I couldn't find them listed.
Linking to the pickle documentation might be a good start.
Will do.
Yes, the execution context is Executor-dependent. The section under
ProcessPoolExecutor and ThreadPoolExecutor spells this out, I think.
I suppose so. I guess I'm just looking for more precise usage of
terminology. (This is a PEP, after all. It's a specification that
multiple VMs may have to follow, not just some user documentation
for a package, even if they'll *probably* be using your code in all
cases.) I'd be happier if there were a clearer term than "calls"
for the things being scheduled ("submissions"?), since the done
callbacks aren't called in the subprocess for ProcessPoolExecutor,
as we just discussed.
Sure. Really, almost any contract would work, it just needs to be
spelled out. It might be nice to know whether the thread invoking
the callbacks is a daemon thread or not, but I suppose it's not
strictly necessary.
Your concerns is that the thread will be killed when the
interpreter exits? It won't be.
Good to know. Tell it to the PEP though, not me ;).
Will do.
No reaction on [invoker vs. future]? I think you'll wish you did
this in a couple of years when you start bumping into application
code that calls "set_result" :).
My reactions are mixed ;-)
Well, you are not obliged to take my advice, as long as I am not
obliged to refrain from mocking you mercilessly if it happens that I
was right in a couple of years ;-).
I was looking for your reasoning rather than trying to negotiate the
circumstances under which you would mock me.
Your proposal is to add a level of indirection to make it harder
for people to call implementation methods. The downside is that it
makes it a bit harder to write tests and Executors.
Both tests and executors will still create and invoke methods
directly on one object; the only additional difficulty seems to be
the need to type '.future' every so often on the executor/testing
side of things, and that seems a cost well worth paying to avoid
confusion over who is allowed to call those methods and when.
I also can't see a big problem in letting people call set_result in
client code though it is documented as being only for Executor
implementations and tests.
On the implementation side, I don't see why an Invoker needs a
reference to the future.
Well, uh...
class Invoker(object):
def __init__(self):
"""Should only be called by Executor implementations."""
self.future = Future()
^ this is what I'd call a "reference to the future"
I said exactly the opposite of what I meant: futures don't need a
reference to the invoker.
Cheers,
Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 26, 2010, at 4:55 AM, Brian Quinlan wrote: > I said exactly the opposite of what I meant: futures don't need a reference > to the invoker. Indeed they don't, and they really shouldn't have one. If I wrote that they did, then it was an error. ... and that appears to be it! Thank you for your very gracious handling of a pretty huge pile of criticism :). Good luck with the PEP, -glyph ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] variable name resolution in exec is incorrect
Hi, issue991196 was closed being described as intentional. I've added a comment in that issue which argues that this is a serious bug (also aserted by a previous commenter - Armin Rigo), because it creates a unique, undocumented, oddly behaving scope that doesn't apply closures correctly. At the very least I think this should be acknowledged as a plain old bug (rather than a feature), and then a discussion about whether it will be fixed or not. Appreciate your thoughts - cheers, Colin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 04:25:18 -0400 Glyph Lefkowitz wrote: > > In other words, while I kinda-sorta buy Brian's argument that having this > module in easy reach > will motivate more people to use a standard, tested idiom for > parallelization, I *don't* think > that the stdlib should be expanded simply to accommodate those who just don't > want to install > additional packages for anything. +1. Why don't the castrated-by-the-corporation people offer to maintain a "Sumo" distribution of Python on python.org instead? The rest of the world shouldn't have to be impacted by their corporate culture woes. cheers Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On Wed, May 26, 2010 at 10:15 AM, Colin H wrote:
> issue991196 was closed being described as intentional. I've added
> a comment in that issue which argues that this is a serious bug (also
> aserted by a previous commenter - Armin Rigo), because it creates a
> unique, undocumented, oddly behaving scope that doesn't apply closures
> correctly. At the very least I think this should be acknowledged as a
> plain old bug (rather than a feature), and then a discussion about
> whether it will be fixed or not.
Here's a quick recap of the issue so that people don't have to go
searching through the bug archive. In Python 2.x, we get the
following behaviour:
>>> code = """\
... y = 3
... def f():
... return y
... f()
... """
>>> exec code in {} # works fine
>>> exec code in {}, {} # dies with a NameError
Traceback (most recent call last):
File "", line 1, in
File "", line 4, in
File "", line 3, in f
NameError: global name 'y' is not defined
The issue is whether the second example should work, given that two
different dictionaries have been passed.
The cause of the NameError can be seen by looking at the bytecode: y
is bound using STORE_NAME, which stores y into the locals dictionary
(which here is *not* the same as the globals dictionary) but the
attempt to retrieve the value of y uses LOAD_GLOBAL, which only looks
in the globals.
>>> co = compile(code, 'mycode', 'exec')
>>> dis.dis(co)
1 0 LOAD_CONST 0 (3)
3 STORE_NAME 0 (y)
2 6 LOAD_CONST 1 ()
9 MAKE_FUNCTION0
12 STORE_NAME 1 (f)
4 15 LOAD_NAME1 (f)
18 CALL_FUNCTION0
21 POP_TOP
22 LOAD_CONST 2 (None)
25 RETURN_VALUE
>>> dis.dis(co.co_consts[1]) # disassembly of 'f'
3 0 LOAD_GLOBAL 0 (y)
3 RETURN_VALUE
This is a long way from my area of expertise (I'm commenting here
because it was me who sent Colin here in the first place), and it's
not clear to me whether this is a bug, and if it is a bug, how it
could be resolved. What would the impact be of having the compiler
produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here?
Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 07:39:15 pm Antoine Pitrou wrote: > On Wed, 26 May 2010 04:25:18 -0400 > > Glyph Lefkowitz wrote: > > In other words, while I kinda-sorta buy Brian's argument that > > having this module in easy reach will motivate more people to use a > > standard, tested idiom for parallelization, I *don't* think that > > the stdlib should be expanded simply to accommodate those who just > > don't want to install additional packages for anything. > > +1. Why don't the castrated-by-the-corporation people offer to > maintain a "Sumo" distribution of Python on python.org instead? The > rest of the world shouldn't have to be impacted by their corporate > culture woes. It's not just the corporate culture. For many people, the standard library is the first introduction to even the existence of a particular technique or technology. You can't go looking for something on PyPI if you don't know that there's a something to look for. And for many beginners and not-so-beginners, the idea and practice of installing additional packages is simply problematic. I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. This is a Python implementation of a stable Java API, Brian has said the futures package has been on PyPI for about a year, and it's been flagged as a production/stable release since October last year. http://pypi.python.org/pypi/futures3 Given that there does seem to be a general agreement that futures should go into the std lib at some point, is this not sufficient exposure? -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Having read through the PEP again, here are my thoughts. * I'm bothered by the term "future". To my mind, it's too long on cleverness and too short on explanativeness. I think that the standard library is no place for cuteness of naming. The name of a stdlib module should reflect its functionality in some straightforward and obvious way. If I were looking for a thread pool or process pool implementation, the word "future" is not something that would spring readily to mind. The stated purpose of the module is to "execute computations asynchronously", so perhaps a name such as "asyntask" would be appropriate, following the pattern of existing modules dealing with ansynchronous matters, ansyncore and asynchat. For the Future object itself, I'd suggest something like "Task" or "Job". * It seems unnecessarily verbose to tack "Executor" onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called "concurrent", I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) Concerning the structure of the PEP: * A section titled 'Specification' should *not* start with a bunch of examples. It may be appropriate to include short examples *following* items in the specification in order to illustrate the features concerned. Extended examples such as these belong in a section of their own. * I found the examples included to be rather difficult to follow, and they served more to confuse than elucidate. I think this is partly because they are written in a fairly compressed style, burying important things being illustrated inside complicated expressions. Rewriting them in a more straightforward style might help. Concerning details of the specification: * Is it possible to have more than one Executor active at a time? The fact that as_completed() is a module-level function rather than an Executor method suggests that it is, but it would be good to have this spelled out one way or the other in the PEP. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 20:42:12 +1000 Steven D'Aprano wrote: > > I'm not saying that Python-Dev should bend over backwards to accommodate > such people to the exclusion of all else, but these folks are > stakeholders too, and their wants and needs are just as worthy as the > wants and needs of those who prefer a more conservative approach to the > standard library. Well, my "Sumo" proposal was a serious one. (not serious in that I would offer to give a hand, but in that I think it could help those people; also, wouldn't it be sensible for users in a corporate environment to share their efforts and produce something that can benefit all of them? it's the free software spirit after all) > This is a Python implementation of a stable Java API, Brian has said the > futures package has been on PyPI for about a year, and it's been > flagged as a production/stable release since October last year. I'm not against futures being in the stdlib, I was just pointing out that I don't agree with the "corporate culture issues should be accomodated by including more modules in the stdlib" argument. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 26, 2010, at 8:57 PM, Greg Ewing wrote: Having read through the PEP again, here are my thoughts. * I'm bothered by the term "future". To my mind, it's too long on cleverness and too short on explanativeness. I think that the standard library is no place for cuteness of naming. The name of a stdlib module should reflect its functionality in some straightforward and obvious way. If I were looking for a thread pool or process pool implementation, the word "future" is not something that would spring readily to mind. The stated purpose of the module is to "execute computations asynchronously", so perhaps a name such as "asyntask" would be appropriate, following the pattern of existing modules dealing with ansynchronous matters, ansyncore and asynchat. For the Future object itself, I'd suggest something like "Task" or "Job". "future" is a computing science term of art, like "thread". Anyway, this has been discussed in the past and Guido was happy with the name. * It seems unnecessarily verbose to tack "Executor" onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. You could have general thread pools that aren't related to executors (actually, it would be great if Python had a good built-in thread pool implementation) and I'd like to avoid using an overly generic name. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called "concurrent", I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. I think that Jesse was planning to add some functionality to this namespace. I don't really have an opinion on this. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) Concerning the structure of the PEP: * A section titled 'Specification' should *not* start with a bunch of examples. It may be appropriate to include short examples *following* items in the specification in order to illustrate the features concerned. Extended examples such as these belong in a section of their own. I thought that the specification would be difficult to follow without examples to pave the way. Anyone else have an opinion on this? * I found the examples included to be rather difficult to follow, and they served more to confuse than elucidate. I think this is partly because they are written in a fairly compressed style, burying important things being illustrated inside complicated expressions. Rewriting them in a more straightforward style might help. Do you think starting with a simpler example would help? I think that idiomatic future use will end up looking similar to my examples. If that is too complex for most users then we have a problem. Concerning details of the specification: * Is it possible to have more than one Executor active at a time? Of course. The fact that as_completed() is a module-level function rather than an Executor method suggests that it is, but it would be good to have this spelled out one way or the other in the PEP. I'll add a note to the global functions that they can accept futures from different in the same call. Cheers, Brian -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 08:57:15 pm Greg Ewing wrote: > * I'm bothered by the term "future". To my mind, it's > too long on cleverness and too short on explanativeness. "Futures" is a standard term in computer science which has been around for 33 years. http://en.wikipedia.org/wiki/Futures_and_promises > I think that the standard library is no place for > cuteness of naming. You mean like pickle, marshal, shelve, turtle, and even dict? > * I don't see a strong reason to put this module > inside a newly-created namespace. If there were a > namespace called "concurrent", I would expect to find > other existing concurrency-related modules there as > well, such as threading and multiprocessing. But we > can't move them there without breaking existing code. I'm sure that it can be done easily, although not quickly. For instance, we move threading into the concurrent namespace, and leave behind in its place a stub: from concurrent.threading import * Then for (say) 3.3 the stub could gain a PendingDeprecation warning, then in 3.4 a Deprecation warning, and finally in 3.5 or 3.6 it could be removed. -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Hi On Sun, May 23, 2010 at 10:47:08AM +1000, Brian Quinlan wrote: > Jesse, the designated pronouncer for this PEP, has decided to keep > discussion open for a few more days. > > So fire away! In thread.py the module automatically registers a handler with atexit. I don't think I'm alone in thinking libraries should not be doing this sort of thing unconditionally behind a user's back. I'm also not so sure how comfortable I am with the module-level globals. Would it not be possible to have an exit handler on each thread pool which the documentation reccomends you register with atexit if it suits your application? I think that would get rid of the global singletons and hidden atexit in a fairly elegant way. Lastly _base.py creates a LOGGER (connected to sys.stderr if I understand correctly) and only logs a critical message to it at the same time as a RuntimeError is raised. While I don't necessarily dislike that it uses a logger, I don't like that it's wired up to sys.stderr I rather think it's the application's duty to create a handler if it wants one. But given that it's only used at the same time as a RuntimeError it does seem redundant. Regards Floris PS: I've only looked at the threading part of the implementation. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010 11:56, Antoine Pitrou wrote: > On Wed, 26 May 2010 20:42:12 +1000 > Steven D'Aprano wrote: >> >> I'm not saying that Python-Dev should bend over backwards to accommodate >> such people to the exclusion of all else, but these folks are >> stakeholders too, and their wants and needs are just as worthy as the >> wants and needs of those who prefer a more conservative approach to the >> standard library. > > Well, my "Sumo" proposal was a serious one. > (not serious in that I would offer to give a hand, but in that I think > it could help those people; also, wouldn't it be sensible for users in > a corporate environment to share their efforts and produce something > that can benefit all of them? it's the free software spirit after all) I'm not sure how a "Sumo" approach would work in practical terms, and this thread isn't really the place to discuss, but there's a couple of points I think are worth making: * For a "Sumo" distribution to make sense, some relatively substantial chunk of the standard library would need to be moved *out* to reside in the sumo distribution. Otherwise it's not really a "sumo", just a couple of modules that "nearly made it into the stdlib", at least for the near-to-medium term. I've yet to see any sort of consensus that python-dev is willing to undertake that decoupling work. (Which would include extracting the various tests, migrating bugs out of the pythion tracker, etc etc). * If the decoupled modules aren't simply being abandoned, python-dev needs to continue to commit to supporting them "in the wild" (i.e., on PyPI and in the sumo distribution). Otherwise we're just abandoning existing users and saying "support it yourself". I've seen no indication that python-dev members would expect to follow bug trackers for various decoupled modules - so in practice, this sounds more like abandonment than decoupling. Until a stdlib-decoupling proposal which takes these aspects into account is on the table, I'm afraid that suggesting there's a "Sumo distribution" style middle ground between stdlib and PyPI isn't really true... Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 8:19 AM, Paul Moore wrote: > On 26 May 2010 11:56, Antoine Pitrou wrote: >> On Wed, 26 May 2010 20:42:12 +1000 >> Steven D'Aprano wrote: >>> >>> I'm not saying that Python-Dev should bend over backwards to accommodate >>> such people to the exclusion of all else, but these folks are >>> stakeholders too, and their wants and needs are just as worthy as the >>> wants and needs of those who prefer a more conservative approach to the >>> standard library. >> >> Well, my "Sumo" proposal was a serious one. >> (not serious in that I would offer to give a hand, but in that I think >> it could help those people; also, wouldn't it be sensible for users in >> a corporate environment to share their efforts and produce something >> that can benefit all of them? it's the free software spirit after all) > > I'm not sure how a "Sumo" approach would work in practical terms, and > this thread isn't really the place to discuss, but there's a couple of > points I think are worth making: > > * For a "Sumo" distribution to make sense, some relatively substantial > chunk of the standard library would need to be moved *out* to reside > in the sumo distribution. Otherwise it's not really a "sumo", just a > couple of modules that "nearly made it into the stdlib", at least for > the near-to-medium term. I've yet to see any sort of consensus that > python-dev is willing to undertake that decoupling work. (Which would > include extracting the various tests, migrating bugs out of the > pythion tracker, etc etc). > > * If the decoupled modules aren't simply being abandoned, python-dev > needs to continue to commit to supporting them "in the wild" (i.e., on > PyPI and in the sumo distribution). Otherwise we're just abandoning > existing users and saying "support it yourself". I've seen no > indication that python-dev members would expect to follow bug trackers > for various decoupled modules - so in practice, this sounds more like > abandonment than decoupling. > > Until a stdlib-decoupling proposal which takes these aspects into > account is on the table, I'm afraid that suggesting there's a "Sumo > distribution" style middle ground between stdlib and PyPI isn't really > true... > > Paul. The fat vs. thin stdlib was discussed on stdlib-sig some time ago (I am generally +1 to having a thin dist and a secondary "fatter" dist), however right now, it doesn't make sense packaging and dependency management is still a mess (but getting better), and there's a ton of other things to take into consideration, some of which has been iterated in this thread. That being said, we've now evolved into meta-meta-meta-discussion - if people seriously want to discuss the fat vs. thin subject, it should probably go to stdlib-sig. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 20:56, Antoine Pitrou wrote: On Wed, 26 May 2010 20:42:12 +1000 Steven D'Aprano wrote: I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. Well, my "Sumo" proposal was a serious one. (not serious in that I would offer to give a hand, but in that I think it could help those people; also, wouldn't it be sensible for users in a corporate environment to share their efforts and produce something that can benefit all of them? it's the free software spirit after all) That's actually what happens with groups like Enthought and ActiveState - bundles with extra batteries. However, note that this isn't just a dysfunctional corporate culture issue, and I object to it being characterised as such (although dysfunctional cultures can certainly make it much, much worse). Vetting licenses for due diligence reasons, tracking releases of an external module, familiarising yourself with an additional API and code base, the risk of encountering bugs in that code base... these are all real costs that don't go away no matter how good the Python packaging ecosystem becomes. There is a trade off between "do the simplest thing that could possibly work (but may cause you problems later)" and spending the time to investigate third party solutions (with the risk that you end up rolling your own later anyway if you don't find anything suitable or, worse, find something that initially seems suitable but proves unworkable in practice). A module that makes it into the standard library, however, carries python-dev's stamp of approval. Except for some older legacy libraries, that means a module will have at least half decent documentation and an automated test suite that is regularly run on multiple platforms. Its design will also have run the gauntlet of python-dev approval. If we identify a good solution to a standard problem, and we have reason to believe that posting it in on PyPI as a separate module won't lead to a significant amount of additional real world testing, then it makes sense for it to go straight into the standard library. Such modules are going to be rare (since most non-trivial modules *will* benefit from some time on PyPI, and most trivial modules won't be added to the standard library at all), but they do exist (runpy, contextlib, collections, itertools and abc spring to mind). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/2010 13:19, Paul Moore wrote: On 26 May 2010 11:56, Antoine Pitrou wrote: On Wed, 26 May 2010 20:42:12 +1000 Steven D'Aprano wrote: I'm not saying that Python-Dev should bend over backwards to accommodate such people to the exclusion of all else, but these folks are stakeholders too, and their wants and needs are just as worthy as the wants and needs of those who prefer a more conservative approach to the standard library. Well, my "Sumo" proposal was a serious one. (not serious in that I would offer to give a hand, but in that I think it could help those people; also, wouldn't it be sensible for users in a corporate environment to share their efforts and produce something that can benefit all of them? it's the free software spirit after all) I'm not sure how a "Sumo" approach would work in practical terms, and this thread isn't really the place to discuss, but there's a couple of points I think are worth making: * For a "Sumo" distribution to make sense, some relatively substantial chunk of the standard library would need to be moved *out* to reside in the sumo distribution. Otherwise it's not really a "sumo", just a couple of modules that "nearly made it into the stdlib", at least for the near-to-medium term. I've yet to see any sort of consensus that python-dev is willing to undertake that decoupling work. (Which would include extracting the various tests, migrating bugs out of the pythion tracker, etc etc). * If the decoupled modules aren't simply being abandoned, python-dev needs to continue to commit to supporting them "in the wild" (i.e., on PyPI and in the sumo distribution). Otherwise we're just abandoning existing users and saying "support it yourself". I've seen no indication that python-dev members would expect to follow bug trackers for various decoupled modules - so in practice, this sounds more like abandonment than decoupling. Until a stdlib-decoupling proposal which takes these aspects into account is on the table, I'm afraid that suggesting there's a "Sumo distribution" style middle ground between stdlib and PyPI isn't really true... Well... a middle ground certainly could exist; perhaps in the form of an "Extended Standard Library" (community distribution), with simple installation and management tools. It could be "blessed" by python-dev and maintain a high standard (only well established best-of-breed modules with a commitment of ongoing maintenance and more than one maintainer - something that the stdlib itself doesn't stick to). A common license could even be chosen, potentially allowing corporations to approve the extended package in a single pass. Lot of details to flesh out obviously - but it would be great to see something like this come into being. Obviously this would need to be a community initiative and would take some time to establish. A "fat" distribution like this, based on tools like pip and distribute would be great for both newbies and for experienced programmers in making it easier to find "best" solutions for standard problems. It could also act as an incubator for the standard library (perhaps with stable and experimental streams where stable has a more conservative update policy). All the best, Michael Foord Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010, at 22:06, Floris Bruynooghe wrote: Hi On Sun, May 23, 2010 at 10:47:08AM +1000, Brian Quinlan wrote: Jesse, the designated pronouncer for this PEP, has decided to keep discussion open for a few more days. So fire away! In thread.py the module automatically registers a handler with atexit. I don't think I'm alone in thinking libraries should not be doing this sort of thing unconditionally behind a user's back. I'm also not so sure how comfortable I am with the module-level globals. Would it not be possible to have an exit handler on each thread pool which the documentation reccomends you register with atexit if it suits your application? I think that would get rid of the global singletons and hidden atexit in a fairly elegant way. First let me explain why I install at atexit handler. Imagine that the you write a script like this: t = ThreadPoolExecutor(1) t.submit(lambda url: print(urllib.open(url).read()), 'http://www.apple.com/') You have two semantic choices here: 1. let the interpreter exit with the future still running 2. wait until the future finishes and then exit I chose (2) but can be convinced otherwise. The obvious way to accomplish this is to make the worker thread non-daemon so the interpreter won't exit while it is running. But since the worker thread is part of a pool, it won't stop while it's executor is alive. So my approach was to make worker threads daemon and install an atexit handler that sets a global indicating that the interpreter is exiting so any workers should exit when when their work queues are empty. It then calls join on each worker thread so the interpreter will not exit until they are finished. I think that this approach is reasonable assuming that you want (2). I also don't have the aversion to globals that you do :-) Lastly _base.py creates a LOGGER (connected to sys.stderr if I understand correctly) and only logs a critical message to it at the same time as a RuntimeError is raised. While I don't necessarily dislike that it uses a logger, I don't like that it's wired up to sys.stderr I rather think it's the application's duty to create a handler if it wants one. But given that it's only used at the same time as a RuntimeError it does seem redundant. The LOGGER is only use for "impossible" exceptions (exceptions in the machinery of the module itself) that won't be propagated because they occur in worker threads. Cheers, Brian Regards Floris PS: I've only looked at the threading part of the implementation. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 17:38, Antoine Pitrou wrote: On Wed, 26 May 2010 09:54:13 +1000 Nick Coghlan wrote: What I would question here is what other things will be part of the "concurrent" package, and who will implement them. Are there plans for that? (or even tracker issues open?) I'm not sure it is called out explicitly in the PEP, but the specific example that came up in the previous discussions was something like "concurrent.pool" to hold a thread vs process agnostic worker pool interface based on the existing Pool interface in multiprocessing (with concrete implementations for both threading and multiprocessing). Ha, I'm a bit surprised. Isn't it what "futures" already provides? (except that for some reason it insists on the "SomeExecutor" naming scheme) http://www.python.org/dev/peps/pep-3148/#processpoolexecutor Not really - a general purpose pool would be a lot more agnostic about how you give the pooled threads/processes work to do and get the results back. Executors are the kind of thing you would build on top of one though. If concurrent.pool was added, then the existing processing pools in multiprocessing and the executors in concurrent.future would be the first use cases for it. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 20:57, Greg Ewing wrote: Having read through the PEP again, here are my thoughts. * It seems unnecessarily verbose to tack "Executor" onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. We would lose the ability to add general purpose thread and process pools under the obvious names later. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called "concurrent", I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. Does it help to know that at least Jesse and I (and probably others) would like to see concurrent.pool added eventually with robust general purpose ThreadPool and ProcessPool implementations? The specific reason the new package namespace was added was to help avoid confusion with stock market futures without using an unduly cumbersome module name, but I don't know how well the PEP explains that. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Sumo
Le mercredi 26 mai 2010 à 13:19 +0100, Paul Moore a écrit : > > I'm not sure how a "Sumo" approach would work in practical terms, and > this thread isn't really the place to discuss, but there's a couple of > points I think are worth making: > > * For a "Sumo" distribution to make sense, some relatively substantial > chunk of the standard library would need to be moved *out* to reside > in the sumo distribution. Otherwise it's not really a "sumo", just a > couple of modules that "nearly made it into the stdlib", at least for > the near-to-medium term. This is not what I'm suggesting at all. The stdlib wouldn't shrink (well, we could dump outdated modules but that's a separate decision). A hypothetical "Sumo" distribution would be made of those more or less useful modules that many people (application or framework developers; for example, take a look at the dozens of direct and indirect dependencies TurboGears pulls in) need and install. The whole point is that it would *not* be supported by python-dev itself but by a separate group of people (such as you :-)). And it would have its own rules (surely more relaxed) over inclusion or deprecation delays, the ability to make compatibility-breaking changes, release timings, multiple obvious ways to do the same thing, etc. And it means it could really bundle a lot of third-party stuff: small helpers (things like the decorator module), event loops, template engines, network address abstractions, argument parsers, ORMs, UI toolkits, etc. (a side-effect would be that it could, if it works well, behave as a good intermediate stress test - a purgatory, if you want - for modules before they integrate the stdlib) If you want an existing analogy, you could look at EasyPHP. Or think of Python as the Gnome or KDE project (consistent and aiming at providing the most important everyday tools, but quite focused), and "Sumo" as an entire distribution of disparate Linux GUI apps. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, 26 May 2010 22:32:33 +1000 Nick Coghlan wrote: > > > > Ha, I'm a bit surprised. Isn't it what "futures" already provides? > > (except that for some reason it insists on the "SomeExecutor" naming > > scheme) > > http://www.python.org/dev/peps/pep-3148/#processpoolexecutor > > Not really - a general purpose pool would be a lot more agnostic about > how you give the pooled threads/processes work to do and get the results > back. > > Executors are the kind of thing you would build on top of one though. If > concurrent.pool was added, then the existing processing pools in > multiprocessing and the executors in concurrent.future would be the > first use cases for it. I think I'm a bit ignorant, but how is the Executor abstraction (and its proposed implementations) not generic enough? You have a pool, submit one or several tasks, and can either repeatedly poll for completion or do a blocking wait. (after all, Glyph pointed out that it should be quite easy to wrap the resulting Futures into Deferred objects) cheers Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 26/05/10 19:48, Mark Dickinson wrote: This is a long way from my area of expertise (I'm commenting here because it was me who sent Colin here in the first place), and it's not clear to me whether this is a bug, and if it is a bug, how it could be resolved. What would the impact be of having the compiler produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here? exec with a single argument = module namespace exec with two arguments = class namespace Class namespaces are deliberately exempted from lexical scoping so that methods can't see class attributes, hence the example in the tracker issue works exactly as it would if the code was written as a class body. class C: y = 3 def execfunc(): print y execfunc() With this code, y would end up in C.__dict__ rather than the module globals (at least, it would if it wasn't for the exception) and the call to execfunc fails with a NameError when attempting to find y. I know I've closed other bug reports that were based on the same misunderstanding, and I didn't understand it myself until Guido explained it to me a few years back, so suggestions for improving the exec documentation in this area would be appreciated. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 8:42 AM, Nick Coghlan wrote: > On 26/05/10 20:57, Greg Ewing wrote: >> >> Having read through the PEP again, here are my thoughts. >> * It seems unnecessarily verbose to tack "Executor" >> onto the end of every Executor subclass. They could >> simply be called ThreadPool and ProcessPool without >> losing anything. > > We would lose the ability to add general purpose thread and process pools > under the obvious names later. > >> * I don't see a strong reason to put this module >> inside a newly-created namespace. If there were a >> namespace called "concurrent", I would expect to find >> other existing concurrency-related modules there as >> well, such as threading and multiprocessing. But we >> can't move them there without breaking existing code. >> >> (More generally, I'm inclined to think that introducing >> a namespace package for a category of modules having >> existing members in the stdlib is an anti-pattern, >> unless it's done during the kind of namespace refactoring >> that we won't get another chance to perform until Py4k.) > > _thread, threading, Queue and multiprocessing do likely belong here, but > moving them isn't likely to be worth the pain. Does it help to know that at > least Jesse and I (and probably others) would like to see concurrent.pool > added eventually with robust general purpose ThreadPool and ProcessPool > implementations? > > The specific reason the new package namespace was added was to help avoid > confusion with stock market futures without using an unduly cumbersome > module name, but I don't know how well the PEP explains that. I assume(d) it's sufficient to link to the mailing list threads where we hashed this out already ;) The namespace serves a few purposes: 1 > We put futures where it makes sense - into a concurrent package. Futures are a concurrency construct; therefore it simply makes sense to put them within a sub package rather on the top level. 2 > We carve out a box to move to, and add other concurrent things, such as generic pools, Actor implementations, etc. See java.util.concurrent. Things within multiprocessing that don't start with P and rhyme with "rocess" can go here too. Admittedly, it's mainly my own long-term vision to see python-core grow more concurrency tidbits - although I don't know too many people who would complain about it. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010, at 22:42, Nick Coghlan wrote: On 26/05/10 20:57, Greg Ewing wrote: Having read through the PEP again, here are my thoughts. * It seems unnecessarily verbose to tack "Executor" onto the end of every Executor subclass. They could simply be called ThreadPool and ProcessPool without losing anything. We would lose the ability to add general purpose thread and process pools under the obvious names later. * I don't see a strong reason to put this module inside a newly-created namespace. If there were a namespace called "concurrent", I would expect to find other existing concurrency-related modules there as well, such as threading and multiprocessing. But we can't move them there without breaking existing code. (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. Does it help to know that at least Jesse and I (and probably others) would like to see concurrent.pool added eventually with robust general purpose ThreadPool and ProcessPool implementations? The specific reason the new package namespace was added was to help avoid confusion with stock market futures without using an unduly cumbersome module name, but I don't know how well the PEP explains that. It doesn't at all. Are these plans formalized anywhere that I can link to? Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On Wed, May 26, 2010 at 11:48, Mark Dickinson wrote:
> On Wed, May 26, 2010 at 10:15 AM, Colin H wrote:
> > issue991196 was closed being described as intentional. I've added
> > a comment in that issue which argues that this is a serious bug (also
> > aserted by a previous commenter - Armin Rigo), because it creates a
> > unique, undocumented, oddly behaving scope that doesn't apply closures
> > correctly. At the very least I think this should be acknowledged as a
> > plain old bug (rather than a feature), and then a discussion about
> > whether it will be fixed or not.
>
> Here's a quick recap of the issue so that people don't have to go
> searching through the bug archive. In Python 2.x, we get the
> following behaviour:
>
> >>> code = """\
> ... y = 3
> ... def f():
> ... return y
> ... f()
> ... """
> >>> exec code in {} # works fine
> >>> exec code in {}, {} # dies with a NameError
> Traceback (most recent call last):
> File "", line 1, in
> File "", line 4, in
> File "", line 3, in f
> NameError: global name 'y' is not defined
>
> The issue is whether the second example should work, given that two
> different dictionaries have been passed.
>
> The cause of the NameError can be seen by looking at the bytecode: y
> is bound using STORE_NAME, which stores y into the locals dictionary
> (which here is *not* the same as the globals dictionary) but the
> attempt to retrieve the value of y uses LOAD_GLOBAL, which only looks
> in the globals.
>
> >>> co = compile(code, 'mycode', 'exec')
> >>> dis.dis(co)
> 1 0 LOAD_CONST 0 (3)
> 3 STORE_NAME 0 (y)
>
> 2 6 LOAD_CONST 1 ( 0xa22b40, file "mycode", line 2>)
> 9 MAKE_FUNCTION0
> 12 STORE_NAME 1 (f)
>
> 4 15 LOAD_NAME1 (f)
> 18 CALL_FUNCTION0
> 21 POP_TOP
> 22 LOAD_CONST 2 (None)
> 25 RETURN_VALUE
> >>> dis.dis(co.co_consts[1]) # disassembly of 'f'
> 3 0 LOAD_GLOBAL 0 (y)
> 3 RETURN_VALUE
>
> This is a long way from my area of expertise (I'm commenting here
> because it was me who sent Colin here in the first place), and it's
> not clear to me whether this is a bug, and if it is a bug, how it
> could be resolved. What would the impact be of having the compiler
> produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here?
It wouldn't matter. The 'f' function only knows about its own namespace
(separate from the surrounding code's local namespace), and the global
namespace. LOAD_NAME is only different from LOAD_GLOBAL in that it looks in
the local namespace first, but in this case the local namespace contains
nothing. Here's what happens:
'exec code in d1, d2' executes code with 'd1' as locals and 'd2' as globals.
Assignment always operates on the local namespace (barring the 'global'
declaration.) The function definition also assigns to the local namespace,
but the created function knows nothing about that local namespace -- it only
cares about its own namespace and the global namespace, 'd1'.
'exec code in d1' does the same thing as 'exec code in d1, d1': it uses the
same dict for the locals and the globals. The execution of the code doesn't
change -- the assignment to 'y' still assigns to the locals, and the 'f'
function still looks it up in globals, but now *they are the same dict*.
Using the same dict for locals and globals is how modules work, as well.
The main confusion here is the fact that 'exec' doesn't generate closures.
(Nobody was ever confused about this behaviour back in Python
2.0-and-earlier! :-) The reason for that is the disconnect between the
compiler and the exec statement: the compiler sees no enclosing function, so
it doesn't generate a closure. The exec statement, because it gets two
different namespaces, then executes it like a function, with a distinct
local namespace.
--
Thomas Wouters
Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 26/05/2010 13:51, Nick Coghlan wrote: On 26/05/10 19:48, Mark Dickinson wrote: This is a long way from my area of expertise (I'm commenting here because it was me who sent Colin here in the first place), and it's not clear to me whether this is a bug, and if it is a bug, how it could be resolved. What would the impact be of having the compiler produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here? exec with a single argument = module namespace exec with two arguments = class namespace Class namespaces are deliberately exempted from lexical scoping so that methods can't see class attributes, hence the example in the tracker issue works exactly as it would if the code was written as a class body. class C: y = 3 def execfunc(): print y execfunc() With this code, y would end up in C.__dict__ rather than the module globals (at least, it would if it wasn't for the exception) and the call to execfunc fails with a NameError when attempting to find y. I know I've closed other bug reports that were based on the same misunderstanding, and I didn't understand it myself until Guido explained it to me a few years back, so suggestions for improving the exec documentation in this area would be appreciated. Your explanation here is very clear. Is this in the documentation? Michael Cheers, Nick. -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 9:01 AM, Brian Quinlan wrote: > > On 26 May 2010, at 22:42, Nick Coghlan wrote: > >> On 26/05/10 20:57, Greg Ewing wrote: >>> >>> Having read through the PEP again, here are my thoughts. >>> * It seems unnecessarily verbose to tack "Executor" >>> onto the end of every Executor subclass. They could >>> simply be called ThreadPool and ProcessPool without >>> losing anything. >> >> We would lose the ability to add general purpose thread and process pools >> under the obvious names later. >> >>> * I don't see a strong reason to put this module >>> inside a newly-created namespace. If there were a >>> namespace called "concurrent", I would expect to find >>> other existing concurrency-related modules there as >>> well, such as threading and multiprocessing. But we >>> can't move them there without breaking existing code. >>> >>> (More generally, I'm inclined to think that introducing >>> a namespace package for a category of modules having >>> existing members in the stdlib is an anti-pattern, >>> unless it's done during the kind of namespace refactoring >>> that we won't get another chance to perform until Py4k.) >> >> _thread, threading, Queue and multiprocessing do likely belong here, but >> moving them isn't likely to be worth the pain. Does it help to know that at >> least Jesse and I (and probably others) would like to see concurrent.pool >> added eventually with robust general purpose ThreadPool and ProcessPool >> implementations? >> >> The specific reason the new package namespace was added was to help avoid >> confusion with stock market futures without using an unduly cumbersome >> module name, but I don't know how well the PEP explains that. > > It doesn't at all. Are these plans formalized anywhere that I can link to? > > Cheers, > Brian Nope; and I don't think we need to worry about it right now. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26 May 2010, at 22:50, Antoine Pitrou wrote: On Wed, 26 May 2010 22:32:33 +1000 Nick Coghlan wrote: Ha, I'm a bit surprised. Isn't it what "futures" already provides? (except that for some reason it insists on the "SomeExecutor" naming scheme) http://www.python.org/dev/peps/pep-3148/#processpoolexecutor Not really - a general purpose pool would be a lot more agnostic about how you give the pooled threads/processes work to do and get the results back. Executors are the kind of thing you would build on top of one though. If concurrent.pool was added, then the existing processing pools in multiprocessing and the executors in concurrent.future would be the first use cases for it. I think I'm a bit ignorant, but how is the Executor abstraction (and its proposed implementations) not generic enough? You have a pool, submit one or several tasks, and can either repeatedly poll for completion or do a blocking wait. (after all, Glyph pointed out that it should be quite easy to wrap the resulting Futures into Deferred objects) Interesting. Executor.submit() return a Future, which might not be useful in some ThreadPool fire-and-forget use cases but having them doesn't seem harmful. Java does take this approach and it gives you a lot more ways to customize the Executor thread pool i.e. the minimum number of threads running, the maximum number, the amount of time that a thread can be idle before it is killed, the queueing strategy to use (e.g. LIFO, FIFO, priority). Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 23:01, Brian Quinlan wrote: _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. Does it help to know that at least Jesse and I (and probably others) would like to see concurrent.pool added eventually with robust general purpose ThreadPool and ProcessPool implementations? The specific reason the new package namespace was added was to help avoid confusion with stock market futures without using an unduly cumbersome module name, but I don't know how well the PEP explains that. It doesn't at all. Are these plans formalized anywhere that I can link to? Just the previous lot of discussions. The main point that should be mentioned in the PEP is that "futures" on its own was ambiguous as to the applicable domain, but "concurrent.futures" was perfectly clear, without causing any readability problems the way a longer name could. Moving the general purpose pools out to their own module was just an example that occurred to us as something else that could go in that package rather than a concrete plan for implementation. Yes, we're setting ourselves up for inevitable questions as to why the existing modules are top level rather than part of this package, but the minimal pain response there would be to link to them from the package documentation with a note along the lines of "for historical reasons, some modules you might reasonably expect to find in this package are instead provided as top level modules". Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 26/05/10 23:08, Michael Foord wrote: On 26/05/2010 13:51, Nick Coghlan wrote: On 26/05/10 19:48, Mark Dickinson wrote: This is a long way from my area of expertise (I'm commenting here because it was me who sent Colin here in the first place), and it's not clear to me whether this is a bug, and if it is a bug, how it could be resolved. What would the impact be of having the compiler produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here? exec with a single argument = module namespace exec with two arguments = class namespace Class namespaces are deliberately exempted from lexical scoping so that methods can't see class attributes, hence the example in the tracker issue works exactly as it would if the code was written as a class body. class C: y = 3 def execfunc(): print y execfunc() With this code, y would end up in C.__dict__ rather than the module globals (at least, it would if it wasn't for the exception) and the call to execfunc fails with a NameError when attempting to find y. I know I've closed other bug reports that were based on the same misunderstanding, and I didn't understand it myself until Guido explained it to me a few years back, so suggestions for improving the exec documentation in this area would be appreciated. Your explanation here is very clear. Is this in the documentation? It isn't actually - I think Thomas may be right that the current exec documentation was largely written prior to the introduction of lexical scoping. So adding something along these lines would probably be a good place to start. Unfortunately, even my explanation above isn't 100% correct - it misses a subtle distinction where lexical scoping will pass through a class definition nested inside a function definition and see the function scope, but the use of strings for exec code means that the scopes of any containing functions are ignored completely. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 23:29, Brian Quinlan wrote: On 26 May 2010, at 22:50, Antoine Pitrou wrote: I think I'm a bit ignorant, but how is the Executor abstraction (and its proposed implementations) not generic enough? You have a pool, submit one or several tasks, and can either repeatedly poll for completion or do a blocking wait. (after all, Glyph pointed out that it should be quite easy to wrap the resulting Futures into Deferred objects) Interesting. Executor.submit() return a Future, which might not be useful in some ThreadPool fire-and-forget use cases but having them doesn't seem harmful. Java does take this approach and it gives you a lot more ways to customize the Executor thread pool i.e. the minimum number of threads running, the maximum number, the amount of time that a thread can be idle before it is killed, the queueing strategy to use (e.g. LIFO, FIFO, priority). I would say it is precisely that extra configurability which separates the executor pools in the PEP implementation from more flexible general purpose pools. It's something to investigate somewhere along the line, but, as Jesse pointed out, not something we need to worry about specifically for this PEP (except as an example of another module that may eventually end up in the concurrent package) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
Thanks for the details on why the observed behaviour occurs - very clear. My only query would be why this is considered correct? Why is it running as a class namespace, when it is not a class? Is there any reason why this is not considered a mistake? Slightly concerned that this is being considered not a bug because 'it is how it is'. A really good reason why you would want to provide a separate locals dictionary is to get access to the stuff that was defined in the exec()'d code block. Unfortunately this use case is broken by the current behaviour. The only way to get the definitions from the exec()'d code block is to supply a single dictionary, and then try to weed out the definitions from amongst all the other globals, which is very difficult if you don't know in advance what was in the code block you exec()'d. So put simply - the bug is that a class namespace is used, but its not a class. On 26/05/2010 13:51, Nick Coghlan wrote: > On 26/05/10 19:48, Mark Dickinson wrote: >> This is a long way from my area of expertise (I'm commenting here >> because it was me who sent Colin here in the first place), and it's >> not clear to me whether this is a bug, and if it is a bug, how it >> could be resolved. What would the impact be of having the compiler >> produce 'LOAD_NAME' rather than 'LOAD_GLOBAL' here? > > exec with a single argument = module namespace > exec with two arguments = class namespace > > Class namespaces are deliberately exempted from lexical scoping so > that methods can't see class attributes, hence the example in the > tracker issue works exactly as it would if the code was written as a > class body. > > class C: > y = 3 > def execfunc(): > print y > execfunc() > > With this code, y would end up in C.__dict__ rather than the module > globals (at least, it would if it wasn't for the exception) and the > call to execfunc fails with a NameError when attempting to find y. > > I know I've closed other bug reports that were based on the same > misunderstanding, and I didn't understand it myself until Guido > explained it to me a few years back, so suggestions for improving the > exec documentation in this area would be appreciated. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 6:56 AM, Antoine Pitrou wrote: > On Wed, 26 May 2010 20:42:12 +1000 > Steven D'Aprano wrote: >> >> I'm not saying that Python-Dev should bend over backwards to accommodate >> such people to the exclusion of all else, but these folks are >> stakeholders too, and their wants and needs are just as worthy as the >> wants and needs of those who prefer a more conservative approach to the >> standard library. > > Well, my "Sumo" proposal was a serious one. > (not serious in that I would offer to give a hand, but in that I think > it could help those people; also, wouldn't it be sensible for users in > a corporate environment to share their efforts and produce something > that can benefit all of them? it's the free software spirit after all) Not in a corporate environment, but I would gladly help with this. Geremy Condra ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sumo
On Wed, May 26, 2010 at 5:46 AM, Antoine Pitrou wrote: > Le mercredi 26 mai 2010 à 13:19 +0100, Paul Moore a écrit : >> >> I'm not sure how a "Sumo" approach would work in practical terms, and >> this thread isn't really the place to discuss, but there's a couple of >> points I think are worth making: >> >> * For a "Sumo" distribution to make sense, some relatively substantial >> chunk of the standard library would need to be moved *out* to reside >> in the sumo distribution. Otherwise it's not really a "sumo", just a >> couple of modules that "nearly made it into the stdlib", at least for >> the near-to-medium term. > > This is not what I'm suggesting at all. The stdlib wouldn't shrink > (well, we could dump outdated modules but that's a separate decision). > > A hypothetical "Sumo" distribution would be made of those more or less > useful modules that many people (application or framework developers; > for example, take a look at the dozens of direct and indirect > dependencies TurboGears pulls in) need and install. > > The whole point is that it would *not* be supported by python-dev itself > but by a separate group of people (such as you :-)). > And it would have its own rules (surely more relaxed) over inclusion or > deprecation delays, the ability to make compatibility-breaking changes, > release timings, multiple obvious ways to do the same thing, etc. > > And it means it could really bundle a lot of third-party stuff: small > helpers (things like the decorator module), event loops, template > engines, network address abstractions, argument parsers, ORMs, UI > toolkits, etc. > > (a side-effect would be that it could, if it works well, behave as a > good intermediate stress test - a purgatory, if you want - for modules > before they integrate the stdlib) > > If you want an existing analogy, you could look at EasyPHP. Or think of > Python as the Gnome or KDE project (consistent and aiming at providing > the most important everyday tools, but quite focused), and "Sumo" as an > entire distribution of disparate Linux GUI apps. > > Regards > > Antoine. I'd also point out that creating a sumo distribution would focus attention on the need to port those libraries which were a part of it to python3, which would help to weaken the argument that there aren't any packages written for it. Geremy Condra ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 26/05/10 20:57, Greg Ewing wrote: (More generally, I'm inclined to think that introducing a namespace package for a category of modules having existing members in the stdlib is an anti-pattern, As a user, I agree with this. unless it's done during the kind of namespace refactoring that we won't get another chance to perform until Py4k.) Is Steven D'Aprano's suggestion (in another post) possible? I'm sure that it can be done easily, although not quickly. For instance, we move threading into the concurrent namespace, and leave behind in its place a stub: from concurrent.threading import * Then for (say) 3.3 the stub could gain a PendingDeprecation warning, then in 3.4 a Deprecation warning, and finally in 3.5 or 3.6 it could be removed. On 5/26/2010 8:42 AM, Nick Coghlan wrote: _thread, threading, Queue and multiprocessing do likely belong here, but moving them isn't likely to be worth the pain. As a user, I disagree and think collecting together these and future modules (pun intended) would be beneficial. There are, from my viewpoint, too many top level modules already. [and in another thread] Yes, we're setting ourselves up for inevitable questions as to why the existing modules are top level rather than part of this package, Yes, forever until they are put in the package. but the minimal pain responsethere For the developers, for the short term would be to link to them from the package documentation with a note along the lines of "for historical reasons, some modules you might reasonably expect to find in this package are instead provided as top level modules". You are, in my opinion, overly discounting the cognitive load and confusion on the part of new users. It would be much better to link *to* subpackage documentation *from* a top level entries (until deleted) and just say that the top level synonyms are present for the obvious historical reason that there once no package, just modules. I am suggesting that if we add a package, we do it right, from the beginning. Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 3:57 AM, Greg Ewing wrote: > Having read through the PEP again, here are my thoughts. > > * I'm bothered by the term "future". To my mind, it's > too long on cleverness and too short on explanativeness. > > I think that the standard library is no place for > cuteness of naming. The name of a stdlib module should > reflect its functionality in some straightforward and > obvious way. If I were looking for a thread pool or > process pool implementation, the word "future" is not > something that would spring readily to mind. Please go re-read the older threads on this. For example, http://mail.python.org/pipermail/python-dev/2010-March/098279.html. > * It seems unnecessarily verbose to tack "Executor" > onto the end of every Executor subclass. They could > simply be called ThreadPool and ProcessPool without > losing anything. +0 > * I don't see a strong reason to put this module > inside a newly-created namespace. If there were a > namespace called "concurrent", I would expect to find > other existing concurrency-related modules there as > well, such as threading and multiprocessing. But we > can't move them there without breaking existing code. Again, please re-read the older thread (which you participated in). For example, http://mail.python.org/pipermail/python-dev/2010-March/098200.html. Jeffrey ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
Mark Dickinson wrote (with interactice prompts removed)
code = """\
y = 3
def f():
return y
. f()
"""
exec code in {} # works fine
exec code in {}, {} # dies with a NameError
Traceback (most recent call last):
File "", line 1, in
File "", line 4, in
File "", line 3, in f
NameError: global name 'y' is not defined
I verified that 3.1 (with exec adjusted to be a function) operates the same.
On 5/26/2010 8:51 AM, Nick Coghlan wrote:
>exec with a single argument = module namespace
>exec with two arguments = class namespace
I verified in 3.1 that indenting 'code' and prepending 'class C():\n'
gives the same error and that prepending 'def f():\n' now, with nexted
function namespaces, does not give an error, although it would have been
an error before Python 2.2, when there were no nested function namespaces.
On 5/26/2010 10:03 AM, Colin H wrote:
Thanks for the details on why the observed behaviour occurs - very
clear. My only query would be why this is considered correct? Why is
it running as a class namespace, when it is not a class?
You are expecting that it run as a function namespace (with post 2.2
nesting), when it is not a function. Why is that any better?
Is there any
reason why this is not considered a mistake? Slightly concerned that
this is being considered not a bug because 'it is how it is'.
In original Python, the snippet would have given an error whether you
thought of it as being in a class or function context, which is how
anyone who knew Python then would have expected. Consistency is not a bug.
When nested function namespaces were introduced, the behavior of exec
was left unchanged. Backward compatibility is not a bug.
A change could have been proposed for 3.x, but I do not remember such a
discussion and expect it would have been rejected. One can get the
nested function behavior by doing what I did in the test mentioned
above. One could easily write a nested_exec function to do the wrapping
automatically.
-
In http://bugs.python.org/issue8824
I suggest that
"In all cases, if the optional parts are omitted, the code is executed
in the current scope. If only globals is provided, it must be a
dictionary, which will be used for both the global and the local
variables. If globals and locals are given, they are used for the
global and local variables, respectively. If provided, locals can be any
mapping object."
be followed by
"If only globals is provided or if onedict is provided as both globals
and locals, the code is executed in a new top-level scope. If different
objects are given as globals and locals, the code is executed as if it
were in a class statement in a new top-level scope."
to make the behavior clearer.
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
The changes to the docs will definitely help in understanding why this behaves as it does. I would like like to take one last stab though at justifying why this behaviour isn't correct - will leave it alone if these arguments don't stack up :) Appreciate the input and discussion. Terry Jan Reedy wrote > You are expecting that it run as a function namespace (with post 2.2 > nesting), when it is not a function. Why is that any better? Because a class namespace (as I see it) was implemented to treat a specific situation - i.e. that functions in classes cannot see class variables. exec() is a far more generic instrument that has no such explicit requirement - i.e. it feels like hijacking an edge case to meet a requirement that doesn't exist. However 'all locals in an enclosing scope are made available in the function namespace' is generally understood as python's generic closure implementation, and would match more effectively the generic nature of the exec() statement. A litmus test for this sort of thing - if you polled 100 knowledgeable python devs who hadn't encountered this problem or this thread and asked if they would expect exec() to run as a class or function namespace, I think you'd struggle to get 1 of them to expect a class namespace. Functions are the more generic construct, and thus more appropriate for the generic nature of exec() (IMHO). It would appear that the only actual requirement not to make locals in an enclosing scope available in a nested function scope is for a class. The situation we are discussing seems have created a similar requirement for exec(), but with no reason. > In original Python, the snippet would have given an error whether you > thought of it as being in a class or function context, which is how > anyone who knew Python then would have expected. Consistency is not a bug. > When nested function namespaces were introduced, the behavior of exec > was left unchanged. Backward compatibility is not a bug. Generally, most other behaviour did change - locals in enclosing scopes *did* become available in the nested function namespace, which was not backward compatible. Why is a special case made to retain consistency and backward compatibility for code run using exec()? It's all python code. Inconsistent backward compatibility might be considered a bug. Cheers, Colin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sumo
On 26 May 2010 13:46, Antoine Pitrou wrote: > This is not what I'm suggesting at all. The stdlib wouldn't shrink > (well, we could dump outdated modules but that's a separate decision). Ah, OK. In that case, I see the argument for a "Sumo" distribution as weak for a different reason - for general use, the standard library is (nearly!) sufficient, and ignoring specialised use cases, there aren't enough generally useful modules to warrant a "Sumo" distribution (you'd essentially be talking about stuff that "nearly made it into the stdlib", and there's not a huge amount of that). Specialised distributions are another matter - I can see a "web stack" distribution comprising your TurboGears example (or should it be Django, or...?). Enthought essentially do that for a "Scientific Python" distribution. There could easily be others. But a general purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical. (Personally, my "essential extras" are pywin32, cx_Oracle and that's about it - futures might make it if it doesn't get into the stdlib, but that's about all). I'm genuinely struggling to see how a Sumo distribution ever comes into being under your proposal. There's no evidence that anyone wants it (otherwise it would have been created by now!!) and until it exists, it's not a plausible "place" to put modules that don't make it into the stdlib. So (unless I'm missing something) your argument seems to be that if enough good stuff is rejected for stdlib inclusion, this will prompt the people who wanted that stuff included to create a sumo distribution, which addresses the "too many dependencies is bad" argument for inclusion in the stdlib. That sounds like a suspiciously circular argument to me... Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sumo
Le mercredi 26 mai 2010 à 23:41 +0100, Paul Moore a écrit : > > But a general > purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical. > (Personally, my "essential extras" are pywin32, cx_Oracle and that's > about it - futures might make it if it doesn't get into the stdlib, > but that's about all). Well, unless you are short on megabytes or on download bandwidth, would you really care to get some modules you never use? (there are probably some in the stdlib too!) > I'm genuinely struggling to see how a Sumo distribution ever comes > into being under your proposal. There's no evidence that anyone wants > it (otherwise it would have been created by now!!) and until it > exists, it's not a plausible "place" to put modules that don't make it > into the stdlib. I don't think all package owners dream of putting their software in the stdlib. It's likely some don't care, and it's likely some would actively refuse it (because e.g. they don't want to lose control). So the suggestion is not to somehow salvage packages which "don't make it into the stdlib", but to build a broader distribution of packages without necessarily having people bow to the constraints of stdlib inclusion. Of course, I agree with you that someone has to do it if they want it to happen :-) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sumo
On Wed, May 26, 2010 at 3:41 PM, Paul Moore wrote: > On 26 May 2010 13:46, Antoine Pitrou wrote: >> This is not what I'm suggesting at all. The stdlib wouldn't shrink >> (well, we could dump outdated modules but that's a separate decision). > > Ah, OK. In that case, I see the argument for a "Sumo" distribution as > weak for a different reason - for general use, the standard library is > (nearly!) sufficient, and ignoring specialised use cases, there aren't > enough generally useful modules to warrant a "Sumo" distribution > (you'd essentially be talking about stuff that "nearly made it into > the stdlib", and there's not a huge amount of that). > > Specialised distributions are another matter - I can see a "web stack" > distribution comprising your TurboGears example (or should it be > Django, or...?). Enthought essentially do that for a "Scientific > Python" distribution. There could easily be others. But a general > purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical. > (Personally, my "essential extras" are pywin32, cx_Oracle and that's > about it - futures might make it if it doesn't get into the stdlib, > but that's about all). I'm not clear, you seem to be arguing that there's a market for many augmented python distributions but not one. Why not just have one that includes the best from each domain? > I'm genuinely struggling to see how a Sumo distribution ever comes > into being under your proposal. There's no evidence that anyone wants > it (otherwise it would have been created by now!!) Everything worth making has already been made? > and until it exists, it's not a plausible "place" to put modules that don't > make it into the stdlib. Of course its implausible to put something somewhere that doesn't exist... until it does. > So (unless I'm missing something) your argument seems > to be that if enough good stuff is rejected for stdlib inclusion, this > will prompt the people who wanted that stuff included to create a sumo > distribution, which addresses the "too many dependencies is bad" > argument for inclusion in the stdlib. That sounds like a suspiciously > circular argument to me... I'd say rather that there are a large number of specialized tools which aren't individually popular enough to be included in Python, but which when taken together greatly increase its utility, and that sumo offers a way to provide that additional utility to python's users without forcing python core devs to shoulder the maintenance burden. Geremy Condra ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
Brian Quinlan wrote: I think that Jesse was planning to add some functionality to this namespace. Even if that happens, the existing threading and multiprocessing modules would remain outside of it. You could have general thread pools that aren't related to executors Yes, but it should be fairly obvious that the ones defined in the futures module have to do with futures. Namespaces are only a honking great idea if you actually let them do the job they're designed for. I thought that the specification would be difficult to follow without examples to pave the way. Well, for me, what happened was that I saw the examples and thought "WTF is going on here?" Then I read the specification to figure out how the examples worked. It might be better to have a tutorial section preceeding the specification section, containing explanation interspersed with examples. > I think that idiomatic future use will end up looking similar to my examples. Maybe, but code written for pedagogical purposes needs to meet a particularly high standard of clarity. Remember that the reader isn't yet familiar with the idioms, so idiomatic code isn't necessarily going to be easy for him to follow. * Is it possible to have more than one Executor active at a time? Of course. That's good, but I don't think that the "of course" is at all obvious, considering that things such as GUI event loops generally can't be mixed easily. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
Mark Dickinson wrote:
code = """\
... y = 3
... def f():
... return y
... f()
... """
exec code in {} # works fine
exec code in {}, {} # dies with a NameError
Seems to me the whole idea of being able to specify
separate global and local scopes for top-level code is
screwy in the first place. Are there any use cases for
it? Maybe the second scope argument to exec() should
be deprecated?
--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 27/05/2010 00:38, Greg Ewing wrote:
Mark Dickinson wrote:
code = """\
... y = 3
... def f():
... return y
... f()
... """
exec code in {} # works fine
exec code in {}, {} # dies with a NameError
Seems to me the whole idea of being able to specify
separate global and local scopes for top-level code is
screwy in the first place. Are there any use cases for
it? Maybe the second scope argument to exec() should
be deprecated?
Sounds good to me, certainly ends the confusion over this undoubtedly
unintuitive behaviour. :-)
Michael
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog
READ CAREFULLY. By accepting and reading this email you agree, on behalf of
your employer, to release me from all obligations and waivers arising from any
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap,
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in perpetuity, without
prejudice to my ongoing rights and privileges. You further represent that you
have the authority to release me from any BOGUS AGREEMENTS on behalf of your
employer.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
Let me quickly jump in before someone actually deletes the feature.
Nick Coghlan and Thomas Wouters had it right; there is still a use
case. Don't kill it -- documenting it better is of course fine.
It *might* be possible to add a closure to the definition of f in the
case where globals != locals, but I doubt that that would be worth it,
and there's probably some code out there that would actually break, so
I'd say this is not a priority.
--Guido
On Wed, May 26, 2010 at 4:33 PM, Michael Foord
wrote:
> On 27/05/2010 00:38, Greg Ewing wrote:
>>
>> Mark Dickinson wrote:
>>
>> code = """\
>>>
>>> ... y = 3
>>> ... def f():
>>> ... return y
>>> ... f()
>>> ... """
>>>
>> exec code in {} # works fine
>> exec code in {}, {} # dies with a NameError
>>
>> Seems to me the whole idea of being able to specify
>> separate global and local scopes for top-level code is
>> screwy in the first place. Are there any use cases for
>> it? Maybe the second scope argument to exec() should
>> be deprecated?
>>
> Sounds good to me, certainly ends the confusion over this undoubtedly
> unintuitive behaviour. :-)
>
> Michael
>
>
> --
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog
>
> READ CAREFULLY. By accepting and reading this email you agree, on behalf of
> your employer, to release me from all obligations and waivers arising from
> any and all NON-NEGOTIATED agreements, licenses, terms-of-service,
> shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure,
> non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have
> entered into with your employer, its partners, licensors, agents and
> assigns, in perpetuity, without prejudice to my ongoing rights and
> privileges. You further represent that you have the authority to release me
> from any BOGUS AGREEMENTS on behalf of your employer.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sumo
On 27/05/10 09:11, geremy condra wrote: Specialised distributions are another matter - I can see a "web stack" distribution comprising your TurboGears example (or should it be Django, or...?). Enthought essentially do that for a "Scientific Python" distribution. There could easily be others. But a general purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical. (Personally, my "essential extras" are pywin32, cx_Oracle and that's about it - futures might make it if it doesn't get into the stdlib, but that's about all). I'm not clear, you seem to be arguing that there's a market for many augmented python distributions but not one. Why not just have one that includes the best from each domain? Because scientists, financial analysts, web designers, etc all have different needs. A targeted distribution like Scientific Python will include nearly all the stuff a scientist is likely to need, but a financial analyst or web designer would find it lacking. As Paul points out, the current size of the set of modules that are sufficiently general purpose and of high enough quality to qualify for python-dev's blessing, but wouldn't be suitable for inclusion in the normal standard library is fairly small. Particular when most developers are able to get sufficiently valuable modules from PyPI if they genuinely need them. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 02:27, Terry Reedy wrote: I am suggesting that if we add a package, we do it right, from the beginning. This is a reasonable point of view, but I wouldn't want to hold up PEP 3148 over it (call it a +0 for the idea in general, but a -1 for linking it to the acceptance of PEP 3148). A separate short PEP proposing a migration plan that could be accepted or rejected independently of PEP 3148 would likely be valuable. E.g. - no change in 2.x (obviously) - add concurrent.* alternate names in 3.x - rearrange documentation in 3.x, with pointers from old names to new names - put a PendingDeprecationWarning on the old names, but otherwise leave them alone indefinitely - add 2to3 fixers to translate from the old names to the new names in import statements Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 7:36 PM, Greg Ewing wrote: > Brian Quinlan wrote: > >> I think that Jesse was planning to add some functionality to this >> namespace. > > Even if that happens, the existing threading and multiprocessing > modules would remain outside of it. Not entirely; once concurrent.* comes into existence, I will seriously begin looking at what we can move out of multiprocessing, into concurrent.* alongside futures. >> You could have general thread pools that aren't related to executors > > Yes, but it should be fairly obvious that the ones defined > in the futures module have to do with futures. Namespaces are > only a honking great idea if you actually let them do the job > they're designed for. concurrent.* is the namespace, futures is the package within the namespace - concurrent.futures is highly descriptive of the items contained therein. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 8:03 PM, Nick Coghlan wrote: > On 27/05/10 02:27, Terry Reedy wrote: >> >> I am suggesting that if we add a package, we do it right, from the >> beginning. > > This is a reasonable point of view, but I wouldn't want to hold up PEP 3148 > over it (call it a +0 for the idea in general, but a -1 for linking it to > the acceptance of PEP 3148). > > A separate short PEP proposing a migration plan that could be accepted or > rejected independently of PEP 3148 would likely be valuable. > > E.g. > - no change in 2.x (obviously) > - add concurrent.* alternate names in 3.x > - rearrange documentation in 3.x, with pointers from old names to new names > - put a PendingDeprecationWarning on the old names, but otherwise leave > them alone indefinitely > - add 2to3 fixers to translate from the old names to the new names in > import statements > > Cheers, > Nick. Agreed; and intended as a different PEP. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 27/05/10 06:07, Colin H wrote: In original Python, the snippet would have given an error whether you thought of it as being in a class or function context, which is how anyone who knew Python then would have expected. Consistency is not a bug. When nested function namespaces were introduced, the behavior of exec was left unchanged. Backward compatibility is not a bug. Generally, most other behaviour did change - locals in enclosing scopes *did* become available in the nested function namespace, which was not backward compatible. Why is a special case made to retain consistency and backward compatibility for code run using exec()? It's all python code. Inconsistent backward compatibility might be considered a bug. Because strings are opaque to the compiler. The lexical scoping has *no idea* what is inside the string, and the exec operation only has 3 things available to it: - the code object compiled from the string - the supplied globals namespace - the supplied locals namespace It isn't a special case, it's the only way it can possible work. Consider a more complex example: def get_exec_str(): y = 3 return "print(y)" exec(get_exec_str()) Should that code work? Or consider this one: def get_exec_str(): y = 3 return "print y" def run_exec_str(str_to_run): y = 5 exec(str_to_run) run_exec_str(get_exec_str()) Should that work? If yes, should it print 3 or 5? Lexical scoping only works for code that is compiled as part of a single operation - the separation between the compilation of the individual string and the code defining that string means that the symbol table analysis needed for lexical scoping can't cross the boundary. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 09:36, Greg Ewing wrote: Brian Quinlan wrote: I think that Jesse was planning to add some functionality to this namespace. Even if that happens, the existing threading and multiprocessing modules would remain outside of it. You could have general thread pools that aren't related to executors Yes, but it should be fairly obvious that the ones defined in the futures module have to do with futures. Namespaces are only a honking great idea if you actually let them do the job they're designed for. futures.ThreadPoolExecutor would likely be refactored to inherit from the mooted pool.ThreadPool. I'd like to leave that option open, and having two classes with the same name from different modules in a single inheritance tree is one of the places where module namespacing still isn't quite as tidy as we might wish. I'd also consider a simple thread pool and an actual executor different things. I'm fine with the longer names, but if I was going to drop a word from the names, it would actually be "Pool" (i.e. ThreadExecutor, ProcessExecutor). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sumo
On Wed, May 26, 2010 at 4:57 PM, Nick Coghlan wrote: > On 27/05/10 09:11, geremy condra wrote: >>> >>> Specialised distributions are another matter - I can see a "web stack" >>> distribution comprising your TurboGears example (or should it be >>> Django, or...?). Enthought essentially do that for a "Scientific >>> Python" distribution. There could easily be others. But a general >>> purpose "Sumo" distribution *on top of* the stdlib? I'm skeptical. >>> (Personally, my "essential extras" are pywin32, cx_Oracle and that's >>> about it - futures might make it if it doesn't get into the stdlib, >>> but that's about all). >> >> I'm not clear, you seem to be arguing that there's a market for many >> augmented python distributions but not one. Why not just have one >> that includes the best from each domain? > > Because scientists, financial analysts, web designers, etc all have > different needs. My point is just that a web designer probably doesn't care if he's got numpy, nor does a mathematician care if he has cherrypy onboard. They only care when the tools they need aren't there, which is where sumo can help. > A targeted distribution like Scientific Python will include nearly all the > stuff a scientist is likely to need, but a financial analyst or web designer > would find it lacking. Seems like the same point as above, I might be missing something. > As Paul points out, the current size of the set of modules that are > sufficiently general purpose and of high enough quality to qualify for > python-dev's blessing, but wouldn't be suitable for inclusion in the normal > standard library is fairly small. Particular when most developers are able > to get sufficiently valuable modules from PyPI if they genuinely need them. Seems like the point is not to focus on the general purpose ones, but rather to include domain or task specific libraries, and libraries that are close to (but not at) the level where they would be considered for inclusion. Geremy Condra ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Thu, 27 May 2010 10:19:50 +1000
Nick Coghlan wrote:
>
> futures.ThreadPoolExecutor would likely be refactored to inherit from
> the mooted pool.ThreadPool.
There still doesn't seem to be reason to have two different thread pool
APIs, though. Shouldn't there be one obvious way to do it?
> I'd also consider a simple thread pool and an actual executor different
> things. I'm fine with the longer names, but if I was going to drop a
> word from the names, it would actually be "Pool" (i.e. ThreadExecutor,
> ProcessExecutor).
To me, ThreadPool looks a lot more obvious than ThreadExecutor
("obvious" in that I can easily find it again, and I don't need to
read some documentation to know what it is).
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
Mark Dickinson wrote: > Seems to me the whole idea of being able to specify > separate global and local scopes for top-level code is > screwy in the first place. Are there any use cases for > it? Maybe the second scope argument to exec() should > be deprecated? It is running as class namespace that makes the argument that there's no use case. I agree - I can't think of any decent use cases for exec() as class namespace - defining functions and classes only works for a subset of function and class definitions However, if exec() ran as function namespace instead, then the locals dictionary will contain all the definitions from the exec()'d code block, and only those definitions. Very useful. This is a major use case for exec() - defining code from strings (e.g. enabling you to store python code in the database), and using it at runtime. It seems to me this must have been the point of locals in the first place. If you just use globals, then your definitions exist amongst a whole bunch of other python stuff, and unless you know in advance what was defined in your code block, its very difficult to extract them. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan wrote:
> On 27/05/10 06:07, Colin H wrote:
>>>
>>> In original Python, the snippet would have given an error whether you
>>> thought of it as being in a class or function context, which is how
>>> anyone who knew Python then would have expected. Consistency is not a
>>> bug.
>>
>>> When nested function namespaces were introduced, the behavior of exec
>>> was left unchanged. Backward compatibility is not a bug.
>>
>> Generally, most other behaviour did change - locals in enclosing
>> scopes *did* become available in the nested function namespace, which
>> was not backward compatible. Why is a special case made to retain
>> consistency and backward compatibility for code run using exec()? It's
>> all python code. Inconsistent backward compatibility might be
>> considered a bug.
>
> Because strings are opaque to the compiler. The lexical scoping has *no
> idea* what is inside the string, and the exec operation only has 3 things
> available to it:
> - the code object compiled from the string
> - the supplied globals namespace
> - the supplied locals namespace
>
> It isn't a special case, it's the only way it can possible work.
>
> Consider a more complex example:
>
> def get_exec_str():
> y = 3
> return "print(y)"
>
> exec(get_exec_str())
>
> Should that code work?
>
> Or consider this one:
>
> def get_exec_str():
> y = 3
> return "print y"
>
> def run_exec_str(str_to_run):
> y = 5
> exec(str_to_run)
>
> run_exec_str(get_exec_str())
>
> Should that work? If yes, should it print 3 or 5?
>
> Lexical scoping only works for code that is compiled as part of a single
> operation - the separation between the compilation of the individual string
> and the code defining that string means that the symbol table analysis
> needed for lexical scoping can't cross the boundary.
Hi Nick,
I don't think Colin was asking for such things. His use case is
clarify by this post of his:
On Wed, May 26, 2010 at 7:03 AM, Colin H wrote:
> A really good reason why you would want to provide a separate locals
> dictionary is to get access to the stuff that was defined in the
> exec()'d code block. Unfortunately this use case is broken by the
> current behaviour. The only way to get the definitions from the
> exec()'d code block is to supply a single dictionary, and then try to
> weed out the definitions from amongst all the other globals, which is
> very difficult if you don't know in advance what was in the code block
> you exec()'d.
I think here's an example of what he's asking for
def define_stuff(user_code):
context = {'FOO': 42, 'BAR': 10**100}
stuff = {}
exec(user_code, context, stuff)
return stuff
In some other place, define_stuff() is called like this:
user_code = """
EXTRA = 1.1
def func(): return FOO * BAR * EXTRA
"""
stuff = define_stuff(user_code)
func = stuff['func']
print(func())
This can't find the EXTRA variable found. (Another example would be
defining a recursive function -- the function can't find itself.)
The alternative (which Colin complains about) is for define_stuff() to
use a single namespace, initialized with the context, and to return
that -- but he's right that in that case FOO and BAR would be exported
as part of stuff, which may not be the intention. (Another solution
would be to put FOO and BAR in the builtins dict -- but that has other
problems of course.)
> So put simply - the bug is that a class namespace is used, but its not a
> class.
Well, really, the bug is that no closure is created even though there
are separate globals and locals. Unfortunately this is because while
at the point where the code is *executed* the two different contexts
are clearly present, at the point where it is *compiled* this is not
the case -- the compiler normally uses syntactic clues to decide
whether to generate code using closures, in particular, the presence
of nested functions. Note that define_stuff() could be equivalently
written like this, where it is more obvious that the compiler doesn't
know about the separate namespaces:
def define_stuff(user_code):
context = {...}
stuff = {}
compiled_code = compile(user_code, "", "exec")
exec(user_code, context, stuff)
return stuff
This is not easy to fix. The best short-term work-around is probably a
hack like this:
def define_stuff(user_code):
context = {...}
stuff = {}
stuff.update(context)
exec(user_code, stuff)
for key in context:
if key in stuff and stuff[key] == context[key]:
del stuff[key]
return stuff
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On Wed, May 26, 2010 at 5:37 PM, Colin H wrote: > Mark Dickinson wrote: > >> Seems to me the whole idea of being able to specify >> separate global and local scopes for top-level code is >> screwy in the first place. Are there any use cases for >> it? Maybe the second scope argument to exec() should >> be deprecated? > > It is running as class namespace that makes the argument that there's > no use case. I agree - I can't think of any decent use cases for > exec() as class namespace - defining functions and classes only works > for a subset of function and class definitions > > However, if exec() ran as function namespace instead, then the locals > dictionary will contain all the definitions from the exec()'d code > block, and only those definitions. Very useful. Hmm... I see your point, but fear that the implementation is tricky, since it would mean every exec'ed block would have to be compiled as if it were a function body, and that probably violates some other invariants. > This is a major use > case for exec() - defining code from strings (e.g. enabling you to > store python code in the database), and using it at runtime. It seems > to me this must have been the point of locals in the first place. > > If you just use globals, then your definitions exist amongst a whole > bunch of other python stuff, and unless you know in advance what was > defined in your code block, its very difficult to extract them. Check the reconstruction of your use case I just posted. I think you re thinking context == globals(), which makes sense. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
Hi Guido,
Thanks for the possible workaround - unfortunately 'stuff' will
contain a whole stack of things that are not in 'context', and were
not defined in 'user_code' - things that python embeds - a (very
small) selection -
{..., 'NameError': , 'BytesWarning':
, 'dict': , 'input':
, 'oct': ,
'bin': , ...}
It makes sense why this happens of course, but upon return, the
globals dict is very large, and finding the stuff you defined in your
user_code amongst it is a very difficult task. Avoiding this problem
is the 'locals' use-case for me. Cheers,
Colin
On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum wrote:
> This is not easy to fix. The best short-term work-around is probably a
> hack like this:
>
> def define_stuff(user_code):
> context = {...}
> stuff = {}
> stuff.update(context)
> exec(user_code, stuff)
> for key in context:
> if key in stuff and stuff[key] == context[key]:
> del stuff[key]
> return stuff
>
> --
> --Guido van Rossum (python.org/~guido)
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sumo
At 11:41 PM 5/26/2010 +0100, Paul Moore wrote: I'm genuinely struggling to see how a Sumo distribution ever comes into being under your proposal. There's no evidence that anyone wants it (otherwise it would have been created by now!!) Actually, sumo distributions *have* been created; it's just that there are so *many* of them: * Linux distributions * BSD & Mac "ports" systems * ActivePython, the Enthought Distribution, etc. * certain distutils extensions that I shall not name When the idea of "sumo" distributions were first being batted around (10-12 years ago?), I'm not sure even *distutils* existed yet, let alone any of this other stuff. I mean, yeah, the port systems and distros existed, but their coverage wasn't nearly as good as it is today. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On Wed, May 26, 2010 at 5:53 PM, Colin H wrote:
> Thanks for the possible workaround - unfortunately 'stuff' will
> contain a whole stack of things that are not in 'context', and were
> not defined in 'user_code' - things that python embeds - a (very
> small) selection -
>
> {..., 'NameError': , 'BytesWarning':
> , 'dict': , 'input':
> , 'oct': ,
> 'bin': , ...}
>
> It makes sense why this happens of course, but upon return, the
> globals dict is very large, and finding the stuff you defined in your
> user_code amongst it is a very difficult task. Avoiding this problem
> is the 'locals' use-case for me. Cheers,
No, if taken literally that doesn't make sense. Those are builtins. I
think you are mistaken that each of those (e.g. NameError) is in stuff
-- they are in stuff['__builtins__'] which represents the built-in
namespace. You should remove that key from stuff as well.
--Guido
> Colin
>
> On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum wrote:
>> This is not easy to fix. The best short-term work-around is probably a
>> hack like this:
>>
>> def define_stuff(user_code):
>> context = {...}
>> stuff = {}
>> stuff.update(context)
>> exec(user_code, stuff)
>> for key in context:
>> if key in stuff and stuff[key] == context[key]:
>> del stuff[key]
>> return stuff
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
Of course :) - I need to pay more attention. Your workaround should do
the trick. It would make sense if locals could be used for this
purpose, but the workaround doesn't add so much overhead in most
situations. Thanks for the help, much appreciated,
Colin
On Thu, May 27, 2010 at 2:05 AM, Guido van Rossum wrote:
> On Wed, May 26, 2010 at 5:53 PM, Colin H wrote:
>> Thanks for the possible workaround - unfortunately 'stuff' will
>> contain a whole stack of things that are not in 'context', and were
>> not defined in 'user_code' - things that python embeds - a (very
>> small) selection -
>>
>> {..., 'NameError': , 'BytesWarning':
>> , 'dict': , 'input':
>> , 'oct': ,
>> 'bin': , ...}
>>
>> It makes sense why this happens of course, but upon return, the
>> globals dict is very large, and finding the stuff you defined in your
>> user_code amongst it is a very difficult task. Avoiding this problem
>> is the 'locals' use-case for me. Cheers,
>
> No, if taken literally that doesn't make sense. Those are builtins. I
> think you are mistaken that each of those (e.g. NameError) is in stuff
> -- they are in stuff['__builtins__'] which represents the built-in
> namespace. You should remove that key from stuff as well.
>
> --Guido
>
>> Colin
>>
>> On Thu, May 27, 2010 at 1:38 AM, Guido van Rossum wrote:
>>> This is not easy to fix. The best short-term work-around is probably a
>>> hack like this:
>>>
>>> def define_stuff(user_code):
>>> context = {...}
>>> stuff = {}
>>> stuff.update(context)
>>> exec(user_code, stuff)
>>> for key in context:
>>> if key in stuff and stuff[key] == context[key]:
>>> del stuff[key]
>>> return stuff
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 00:31, Brian Quinlan wrote: You have two semantic choices here: 1. let the interpreter exit with the future still running 2. wait until the future finishes and then exit I'd go for (1). I don't think it's unreasonable to expect a program that wants all its tasks to finish to explicitly wait for that to happen. Also, automatically doing (2) would seem to make it difficult for a program to bail out if something unexpected happens. It would have to explicitly shut down the thread pool instead of just letting an exception propagate. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sumo
>
> > Because scientists, financial analysts, web designers, etc all have
> > different needs.
>
> My point is just that a web designer probably doesn't care if he's
> got numpy, nor does a mathematician care if he has cherrypy
> onboard. They only care when the tools they need aren't there,
> which is where sumo can help.
Why do we want "distributions" (whether 'sumo' or domain specific) in the
first place? Obviously because we have some pains to alleviate, but I think
Linux distributions, particularly Ubuntu, have been coping with similar
pains for a while now with good results. I used to be rather informed on the
state of Linux distributions circa the end of the 90's. Should I use this
distribution or that one? Now I don't care, the answer is always 'Ubuntu,
and apt-get it to fit your needs'.
The pain list, as I see it:
(a) Make it easy to find functionality that you'd typically be tempted to
implement alone
(b) Make it trivial, mind boggling easy, to add this functionality to your
interpreter once you know of it
(b.1) Make it easy to package a Python 'distribution' so end users don't
need to muck around with (b)
(c) Make it easy for software to specify its requirements in code and get
them mostly automagically
(d) Make it *legally* least painful to acquire and use endorsed
functionality sets
... (pipe in if I missed something important)
I'm wondering if expanding the efforts in Python packaging ("easy_install"
and PEP376 comes to mind, only on steroids) isn't the correct solution for
alleviating these pains in a much better fashion. I'm not saying it's easy,
but I think that superb packaging technologies are a very well proven
technology, that they're well within the community's capabilities/needs and
that maximizing flexibility is a move in a better direction than creating
one or two or seven 'distributions'.
I hope you would intuitively grasp what I mean, even if I fail to articulate
it well, if you ever used dpkg and apt-get (or equivalents), along with the
ability to add sources and the simple distinctions between sources (core vs.
universe vs. multiverse), along with eco systems like launchpad and PPAs,
etc. To my 1999 self these features of Ubuntu seem like miracles, not so
much because Debian and dpkg/apt-get weren't there back then (they were life
saving back then too), but because of the whole system's huge support base
and centrally-designed decentrally-implemented nature. 'apt-cache search' is
usually how I find new software these days, and the same could be true for
Python when I need futures or multiprocessing or whatever. I could write
lots about what I think we should have in such a system, but maybe it's more
befitting for python-ideas or a blog post than here; for this argument,
'mirror Ubuntu's winnage' is a good enough description. I think a good and
lively packaging system beats 'distributions' any day.
This leaves us only with the legal pain, which I felt harshly on my flesh
following some acquisition by a corporation and it's painful and frustrating
and mostly unreasonable. I suspect, though I'm a legal n00b (and proud of
it), that packaging in multiple repositories like Ubuntu does could help a
lot, along with taking some responsibility over the most recommended repo
(candidates are the 'almost made it into stdlib' packages, maybe even the
giants like Django, NumPy, TurboGears, Twisted, etc). This is not about
development responsibility and surely not about taking legal liability, it's
about actively helping facilitate these packages' use in hostile legal
environments. While the hackers in the trenches working on trunk and py3k
may not care much (and probably needn't care much), I humbly think the PSF
may care more and is best equipped to help. Maybe by requiring certain
licenses for 'recommended' repo inclusion, maybe by working with the likes
of Google or IBM on publishing legal reviews of select Python distributions,
I don't know, depends what would help a corporate lawyer make speedy
approval.
- Yaniv
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
> > Well... a middle ground certainly could exist; perhaps in the form of an > "Extended Standard Library" (community distribution), with simple > installation and management tools. > > It could be "blessed" by python-dev and maintain a high standard (only well > established best-of-breed modules with a commitment of ongoing maintenance > and more than one maintainer - something that the stdlib itself doesn't > stick to). A common license could even be chosen, potentially allowing > corporations to approve the extended package in a single pass. > I read the 'sumo' thread before I read this (and replied in depth there), but I think Michael and I mean similar things. - Yaniv ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Wed, May 26, 2010 at 7:15 PM, Yaniv Aknin wrote: >> Well... a middle ground certainly could exist; perhaps in the form of an >> "Extended Standard Library" (community distribution), with simple >> installation and management tools. I'm not sure about the 'installation and management tools' part, but this is basically the idea I was trying to articulate: a middle ground between a 'fat' stdlib and a 'lean' one. >> It could be "blessed" by python-dev and maintain a high standard (only >> well established best-of-breed modules with a commitment of ongoing >> maintenance and more than one maintainer - something that the stdlib itself >> doesn't stick to). A common license could even be chosen, potentially >> allowing corporations to approve the extended package in a single pass. If we could do it that would be great, IMHO. > I read the 'sumo' thread before I read this (and replied in depth there), > but I think Michael and I mean similar things. > - Yaniv I don't think I'm understanding you correctly in that thread then, ISTM that you're advocating better packaging systems as an alternative to this. Would you mind clarifying? Geremy Condra ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 01:48, Nick Coghlan wrote: I would say it is precisely that extra configurability which separates the executor pools in the PEP implementation from more flexible general purpose pools. Wouldn't this be better addressed by adding the relevant options to the futures pools, rather than adding another module that provides almost exactly the same thing with different options? -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 5/26/2010 8:03 PM, Nick Coghlan wrote: > On 27/05/10 02:27, Terry Reedy wrote: >> I am suggesting that if we add a package, we do it right, from the >> beginning. > > This is a reasonable point of view, but I wouldn't want to hold up PEP > 3148 over it (call it a +0 for the idea in general, but a -1 for linking > it to the acceptance of PEP 3148). That sounds backward. How can you justify accepting PEP 3148 into a "concurrent" namespace without also accepting the demand for such a namespace? What is the contingency if this TBD migration PEP is not accepted, what happens to PEP 3148? After all, there was some complaints about just calling it "futures", without putting it in a "concurrent" namespace. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 27/05/10 11:33, Michael Foord wrote: On 27/05/2010 00:38, Greg Ewing wrote: Maybe the second scope argument to exec() should be deprecated? Sounds good to me, certainly ends the confusion over this undoubtedly unintuitive behaviour. :-) Although it's a fair point that it can be useful to have a way of capturing definitions made by the execed code, while still making an environment of other stuff available to it. So, I'd be in favour of changing the behaviour of exec so that the local scope is made visible inside functions. However, it would be non-trivial to implement this the way things are currently structured, which is probably one of the reasons it hasn't already been done. I don't think that simply using LOAD_NAME inside the function would work, because that would only look in the function's own local namespace first, then in the global one. There is just no mechanism available at the moment for the function to know about the local namespace passed in to exec. The way that functions get access to names in enclosing local scopes is by having them passed in as cells, but that mechanism is only available for optimised local namespaces, not ones implemented as dicts. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 27/05/10 12:04, Jesse Noller wrote: Namespaces are only a honking great idea if you actually let them do the job they're designed for. concurrent.* is the namespace, futures is the package within the namespace - concurrent.futures is highly descriptive of the items contained therein. I was referring to the issue of ThreadPool vs. ThreadPoolExecutor etc. By your own argument above, concurrent.futures.ThreadPool is quite descriptive enough of what it provides. It's not a problem if some other module also provides something called a ThreadPool. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 27/05/10 12:37, Colin H wrote: This is a major use case for exec() - defining code from strings (e.g. enabling you to store python code in the database), and using it at runtime. It seems to me this must have been the point of locals in the first place. I suspect that originally it just fell out of the implementation. The function in the interpreter that executes code objects requires two namespaces as arguments, and they were both exposed via exec just in case anyone found a use for them. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 27/05/10 12:38, Guido van Rossum wrote: the compiler normally uses syntactic clues to decide whether to generate code using closures, in particular, the presence of nested functions. Well, the compiler could be passed a flag indicating that the code is being compiled for an exec statement. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
Another approach to all this might be to generalise the mechanism by which a lookup of the globals falls back to a lookup of __builtins__. If this were done recursively, then the "stuff" could be attached to the globals dict, e.g. stuff['__builtins__'] = __builtins__ g = dict(__builtins__ = stuff) exec(code, g) del g['__builtins__'] -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
> > I don't think I'm understanding you correctly in that thread then, ISTM > that you're advocating better packaging systems as an alternative to > this. Would you mind clarifying? Gladly. In my mind, 'better packaging' is not "just" about something that will let you do 'pypkg install foo' and 'pypkg remove foo' along with dependancies, it's also about meta-packages (packages that have nothing but dependencies) but also about separate repositories with different natures, some endorsed by python-dev, some not. It's not so much the packaging system that does the trick - it's the eco system around it. When you install something from the 'recommended' repository (should find a better name), you know its a 'blessed' package. You know it has unittests, documentation, active development, good license, a bug tracker and the general-soundedness which I hope users have come to expect of python-dev. It is not stdlib, it's not made by python-dev, it doesn't come with python.org's default build by default, but you have a lot of important assurances with ease. You know 'it could have been in stdlib, but happens not to be'. To make this really work, the future tool I called "pypkg" will come with all future Python versions (so people can rely on it), and will arrive configured by default to use only the 'recommended' repository. From a developer's standpoint, relying on package 'foo' of the 'recommended' repository is safe: even if their end users don't happen to have 'foo', the developer is certain that 'recommended' is configured on their user's Python and is a quick (automagical?) download away. - Yaniv (p.s.: again, think Ubuntu core vs universe vs multiverse vs. non-Ubuntu repositories vs. PPAs; forgive me for waving the Ubuntu flag, from the packaging systems /and accompanying eco-systems/ I know, they did it best) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sumo
Hello, sorry to interrupt your discussion but.. On Thu, May 27, 2010 at 04:09, Yaniv Aknin wrote: >> > Because scientists, financial analysts, web designers, etc all have >> > different needs. >> >> My point is just that a web designer probably doesn't care if he's >> got numpy, nor does a mathematician care if he has cherrypy >> onboard. They only care when the tools they need aren't there, >> which is where sumo can help. > > Why do we want "distributions" (whether 'sumo' or domain specific) in the > first place? Obviously because we have some pains to alleviate, but I think > Linux distributions, particularly Ubuntu, have been coping with similar > pains for a while now with good results. I used to be rather informed on the > state of Linux distributions circa the end of the 90's. Should I use this > distribution or that one? Now I don't care, the answer is always 'Ubuntu, > and apt-get it to fit your needs'. please note that the huge bunch of work for Python third-party modules is done in *Debian* and Ubuntu just takes those packages without advertise enough where they come from and who did the work (and not the merchandizing only). > I hope you would intuitively grasp what I mean, even if I fail to articulate > it well, if you ever used dpkg and apt-get (or equivalents), along with the > ability to add sources and the simple distinctions between sources (core vs. > universe vs. multiverse), along with eco systems like launchpad and PPAs, > etc. To my 1999 self these features of Ubuntu seem like miracles, not so > much because Debian and dpkg/apt-get weren't there back then (they were life > saving back then too), but because of the whole system's huge support base > and centrally-designed decentrally-implemented nature. mh? Debian was not in present in 1999? Debian started in 1993 (dpkg in 1994 and apt-get in 1998) while ubuntu only in mid-2000 as a layer over Debian packages (and hiring several Debian Developers for its core positions). Also, let me remind you that transition to Python 2.6 was a huge mess for Ubuntu, where several packages were just left broken (f.e. failing unit tests were disabled to make the package build...) and only now that Debian started to migrate to 2.6 too, you can see a "flow" of packages that works for 2.6 too coming to Ubuntu. Debian can be slow, but we care about quality. End of "give credit where it's due" post :) Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 27, 2010, at 1:21 PM, Greg Ewing wrote: On 27/05/10 12:04, Jesse Noller wrote: Namespaces are only a honking great idea if you actually let them do the job they're designed for. concurrent.* is the namespace, futures is the package within the namespace - concurrent.futures is highly descriptive of the items contained therein. I was referring to the issue of ThreadPool vs. ThreadPoolExecutor etc. By your own argument above, concurrent.futures.ThreadPool is quite descriptive enough of what it provides. It's not a problem if some other module also provides something called a ThreadPool. I think that the "Executor" suffix is a good indicator of the interface being provided. "Pool" is not because you can could have Executor implementations that don't involve pools. Cheers, Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
