Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On May 29, 2013 1:09 AM, "Nick Coghlan" wrote: > > On Wed, May 29, 2013 at 10:14 AM, Brett Cannon wrote: > >> (FWIW, I think "ModuleManager" is a rather bad name :-) > > > > I'm open to suggestions, but the thing does manage the module so it at > > least makes sense. > > I suggest ModuleInitialiser as the CM name, with a helper function to > make usage read better: > > with initialise_module(name) as m: > # Module initialisation code goes here > # Module is rolled back if initialisation fails But you're not initializing the module; more like getting the module, either new or from sys.modules. But I thought ModuleGetter seemed too Java-like. Could hide the class behind a get_module function though. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Structural cleanups to the main CPython repo
On Wed, May 29, 2013 at 12:19 PM, Nick Coghlan wrote: > Anway, I'll come up with some specific patches and put them on the > tracker, starting with moving the source files for the binary > executables and making the simpler pythonrun/lifecycle split. I can > look into splitting lifecycle.c into separate bootstrap and shutdown > translation units after those less controversial changes have been > reviewed (the split may not even be all that practical outside the PEP > 432 branch, since it would involve exposing quite a few currently > static variables to the linker). I started with the simplest part, adding a new Programs directory: http://bugs.python.org/issue18093 Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] decoding setuptools entry point scripts (was: Bilingual scripts)
On Tue, 28 May 2013 22:20:33 -0400, Tres Seaver wrote: > > So, my point is that the information on what python code is actually > > being called ought to be in the stub script file, as a comment if > > nothing else, for discoverability reasons. > > > > I'm not bothered enough to work up a patch, though :) > > It is there already: > > # EASY-INSTALL-ENTRY-SCRIPT: 'pip==1.3.1','console_scripts','pip' > > Which says, load the entry point named 'pip' from the 'console_scripts' > entry point group in the 'pip 1.3.1' distribution. > > The 'entry_points.txt' metadata file specifies that that entry point is a > function named 'main' inside the 'pip' package itself. Ah, but you had to *decode* that for me, using your non-local expert's knowledge. I assume 'main' is defined in or imported into pip's __init__? Now, if the comment had said: # Call pip.main (per the specification in the pip entry of the # console_scripts section of pip-1.3.1-egg-info/entrypoints.txt). then I would have known everything I needed to know without either consulting the *implementor's* documentation for setuptools or an expert such as yourself. Of that, as a *user*, the first two words are the only thing I'm interested in, but the other information could be handy in debugging certain specialized and unlikely issues, such as when someone has manually changed the entrypoints.txt file. Note that the comment still requires you to know python import semantics...but if you don't know that much you wouldn't get far looking at the source code anyway. The dir/filename lets you 'find' the entrypoints.txt file even if you don't know where on your system that file is installed. --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, May 29, 2013 at 11:04 PM, Brett Cannon wrote: >> with initialise_module(name) as m: >> # Module initialisation code goes here >> # Module is rolled back if initialisation fails > > But you're not initializing the module; more like getting the module, either > new or from sys.modules. But I thought ModuleGetter seemed too Java-like. > Could hide the class behind a get_module function though. The point is to provide a useful mnemonic for *why* you would use this context manager, and the reason is because the body of the with statement is going to initialize the contents, and you want to unwind things appropriately if that fails. initializing_module is probably a better name than initialized_module, though (since it isn't initialized yet on entry - instead, that's what should be the case by the end of the statement) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, May 29, 2013 at 10:34 AM, Nick Coghlan wrote: > On Wed, May 29, 2013 at 11:04 PM, Brett Cannon wrote: >>> with initialise_module(name) as m: >>> # Module initialisation code goes here >>> # Module is rolled back if initialisation fails >> >> But you're not initializing the module; more like getting the module, either >> new or from sys.modules. But I thought ModuleGetter seemed too Java-like. >> Could hide the class behind a get_module function though. > > The point is to provide a useful mnemonic for *why* you would use this > context manager, and the reason is because the body of the with > statement is going to initialize the contents, and you want to unwind > things appropriately if that fails. You should use this context manager to get the correct module to initialize/execute/whatever, e.g. contextlib.closing is about what the context manager is going to do for you, not what you are doing to the object it returned. > > initializing_module is probably a better name than initialized_module, > though (since it isn't initialized yet on entry - instead, that's what > should be the case by the end of the statement) I am willing to compromise to module_to_initialize, module_to_init, or module_to_load. Pick one. =) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Thu, May 30, 2013 at 12:47 AM, Brett Cannon wrote: > I am willing to compromise to module_to_initialize, module_to_init, or > module_to_load. Pick one. =) I see this as *really* similar to a database transaction, and those start with "session.begin()". Could you tolerate "with begin_module_init(name) as m:"? We could even document the ability to check m.__initializing__ to see whether this is a reload() or not. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Thu, 30 May 2013 00:59:02 +1000, Nick Coghlan wrote: > On Thu, May 30, 2013 at 12:47 AM, Brett Cannon wrote: > > I am willing to compromise to module_to_initialize, module_to_init, or > > module_to_load. Pick one. =) > > I see this as *really* similar to a database transaction, and those > start with "session.begin()". > > Could you tolerate "with begin_module_init(name) as m:"? But for a transaction, it is 'with session', not 'with begin_session'. With 'begin_module_init' I would have no idea what 'm' was. With Brett's 'module_to_init' I have an intuitive idea about what 'm' is. And if 'm' isn't a module, then module_manager would be better. (Note that I haven't grokked what Brett's context manager is actually doing/returning, I'm speaking here as an ignorant reader of someone else's code :) > We could even document the ability to check m.__initializing__ to see > whether this is a reload() or not. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
Le Wed, 29 May 2013 11:58:21 -0400, "R. David Murray" a écrit : > On Thu, 30 May 2013 00:59:02 +1000, Nick Coghlan > wrote: > > On Thu, May 30, 2013 at 12:47 AM, Brett Cannon > > wrote: > > > I am willing to compromise to module_to_initialize, > > > module_to_init, or module_to_load. Pick one. =) > > > > I see this as *really* similar to a database transaction, and those > > start with "session.begin()". > > > > Could you tolerate "with begin_module_init(name) as m:"? > > But for a transaction, it is 'with session', not 'with begin_session'. or "with transaction.begin()", or "with transaction.commit_on_success()", depending on the API :-) > With 'begin_module_init' I would have no idea what 'm' was. With > Brett's 'module_to_init' I have an intuitive idea about what 'm' is. Agreed. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, May 29, 2013 at 11:58 AM, R. David Murray wrote: > On Thu, 30 May 2013 00:59:02 +1000, Nick Coghlan wrote: >> On Thu, May 30, 2013 at 12:47 AM, Brett Cannon wrote: >> > I am willing to compromise to module_to_initialize, module_to_init, or >> > module_to_load. Pick one. =) >> >> I see this as *really* similar to a database transaction, and those >> start with "session.begin()". >> >> Could you tolerate "with begin_module_init(name) as m:"? > > But for a transaction, it is 'with session', not 'with begin_session'. > > With 'begin_module_init' I would have no idea what 'm' was. With > Brett's 'module_to_init' I have an intuitive idea about what 'm' is. > And if 'm' isn't a module, then module_manager would be better. > > (Note that I haven't grokked what Brett's context manager is actually > doing/returning, I'm speaking here as an ignorant reader of someone > else's code :) In case you want to suggest a name, the context manager returns the module that should be initialized/loaded. So typical usage will be:: class Loader: def load_module(self, fullname): with importlib.util.module_to_init(fullname) as module: # Load/initialize the module return module Basically the manager either gets the module from sys.modules if it is already there (a reload) or creates a new module and sticks it into sys.modules so other imports will grab the right module object. If there is an exception and the module was new, it deletes it from sys.modules to prevent stale modules from sticking around. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, 29 May 2013 12:25:45 -0400, Brett Cannon wrote: > In case you want to suggest a name, the context manager returns the > module that should be initialized/loaded. So typical usage will be:: > > class Loader: > def load_module(self, fullname): > with importlib.util.module_to_init(fullname) as module: > # Load/initialize the module > return module > > Basically the manager either gets the module from sys.modules if it is > already there (a reload) or creates a new module and sticks it into > sys.modules so other imports will grab the right module object. If > there is an exception and the module was new, it deletes it from > sys.modules to prevent stale modules from sticking around. So it is a context manager to handle the exception? Now I think I see where Nick is coming from. How about 'managed_initializiation'? That seems closer to the 'closing' model, to me. It isn't as clear about what it is returning, though, since you are passing it a name and getting back a module, whereas in the closing case you get back the same object you send in. Perhaps 'managed_module'? --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, May 29, 2013 at 12:49 PM, R. David Murray wrote: > On Wed, 29 May 2013 12:25:45 -0400, Brett Cannon wrote: >> In case you want to suggest a name, the context manager returns the >> module that should be initialized/loaded. So typical usage will be:: >> >> class Loader: >> def load_module(self, fullname): >> with importlib.util.module_to_init(fullname) as module: >> # Load/initialize the module >> return module >> >> Basically the manager either gets the module from sys.modules if it is >> already there (a reload) or creates a new module and sticks it into >> sys.modules so other imports will grab the right module object. If >> there is an exception and the module was new, it deletes it from >> sys.modules to prevent stale modules from sticking around. > > So it is a context manager to handle the exception? It's to choose the right module (sys.modules or new) and if the module is new and there is an exception to delete it from sys.modules (other small details like setting __initializing__ but that's not important). So both __enter__ and __exit__ have logic. > Now I think I see > where Nick is coming from. > > How about 'managed_initializiation'? That seems closer to the 'closing' > model, to me. It isn't as clear about what it is returning, though, > since you are passing it a name and getting back a module, whereas in > the closing case you get back the same object you send in. True. > > Perhaps 'managed_module'? managed_module is better than managed_initialization. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] performance testing recommendations in devguide
The devguide doesn't have anything on performance testing that I could find. We do have a number of relatively useful resources in this space though, like pybench and (eventually) speed.python.org. I'd like to add a page to the devguide on performance testing, including an explanation of our performance goals, how to test for them, and what tools are available. Tools I'm aware of: * pybench (relatively limited in real-world usefulness) * timeit module (for quick comparisions) * benchmarks repo (real-world performance test suite) * speed.python.org (would omit for now) Things to test: * speed * memory (tools? tests?) Critically sensitive performance subjects * interpreter start-up time * module import overhead * attribute lookup overhead (including MRO traversal) * function call overhead * instance creation overhead * dict performance (the underlying namespace type) * tuple performance (packing/unpacking, integral container type) * string performance What would be important to say in the devguide regarding Python performance and testing it? What would you add/subtract from the above? How important is testing memory performance? How do we avoid performance regressions? Thanks! -eric ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, 29 May 2013 12:55:01 -0400 Brett Cannon wrote: > > Perhaps 'managed_module'? > > managed_module is better than managed_initialization. I don't understand how it's "managed". "manage", "manager", etc. is the kind of dumb words everybody uses when they don't manage (!) to explain what they're talking about. My vote is for "module_to_init", "uninitialized_module", "pristine_module", etc. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bilingual scripts
On May 28, 2013, at 08:02 PM, Antoine Pitrou wrote:
>On Tue, 28 May 2013 13:27:18 -0400
>Barry Warsaw wrote:
>> On May 25, 2013, at 09:53 AM, Antoine Pitrou wrote:
>>
>> >How about always running the version specific targets, e.g.
>> >nosetests-2.7?
>>
>> We have nosetests-2.7 and nosetests3 in /usr/bin, but we generally recommend
>> folks not use these, especially for things like (build time) package tests.
>> It's harder to iterate over when the installed versions are unknown
>> statically, e.g. if you wanted to run all the tests over all available
>> versions of Python.
>
>It sounds like you want a dedicated script or utility for this ("run
>all the tests over all available versions of Python") rather than hack
>it every time you package a Python library.
There is some support for this in some of the Debian helpers, e.g. pybuild,
but that's not in widespread use. One problem is that there's no definitive
way to know how to run a package's test suite (and won't be even in after PEP
426). I tried to generate some momentum around trying to standardize this,
but it didn't get anywhere.
Still, that's just one small aspect of the problem.
>Your use case also doesn't seem to impact end-users.
Depends on who the end-users are. It definitely impacts developers.
>> This is why I would really like to see all scripts provide a -m equivalent
>> for command line invocation. This might be a little awkward for < Python
>> 2.7 (where IIRC -m doesn't work with packages).
>
>Do you still support Python < 2.7?
Debian still does support 2.6, but hopefully not for long!
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bilingual scripts
On May 28, 2013, at 12:23 PM, Toshio Kuratomi wrote: >> Note that the Gentoo example also takes into account versions that might act >> differently based on the interpreter's implementation. So a -python3 suffix >> may not be enough. Maybe now we're getting into PEP 425 compatibility tag >> territory. >> > This is an interesting, unmapped area in Fedora at the moment... I >was hoping to talk to Nick and the Fedora python maintainer at our next >Fedora conference. > >I've been looking at how Fedora's ruby guidelines are implemented wrt >alternate interpreters and wondering if we could do something similar for >python: > >https://fedoraproject.org/wiki/Packaging:Ruby#Different_Interpreters_Compatibility Very interesting. It was something like this, albeit replacing _jruby_ or _mri_ with something like --py2 or --py3 that I had in mind. However... >I'm not sure yet how much of that I'd (or Nick and the python maintainer >[bkabrda, the current python maintainer is the one who wrote the rubypick >script]) would want to use in python -- replacing /usr/bin/python with a >script that chooses between CPython and pypy based on user preference gave >me an instinctual feeling of dread the first time I looked at it but it >seems to be working well for the ruby folks. ... it kind of gives me the heebie-jeebies too. I think *most* scripts wouldn't need this kind of variability though. For example, lsb_release only needs to run with one version of Python, so: % head -1 /usr/bin/lsb_release #! /usr/bin/python3 -Es is just fine. I wouldn't want to replace /usr/bin/python with a selectable interpreter (see also PEP 394), but if we had something like /usr/bin/multipy which acted like rubypick for the few, very limited examples where it's needed, then it might be useful to do so. I would very definitely want to get consensus on the mechanism and api between the various Linux distros here so it works the same on F/RH, D/U, Gentoo and any others. >My current feeling is that I wouldn't use this same system for interpreters >which are not mostly compatible (for instance, python2 vs python3). but I >also haven't devoted much actual time to thinking about whether that might >have some advantages. Seems like for Python, that would be the most important use case, but maybe I have blinders on for the issue at hand. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bilingual scripts
On May 29, 2013, at 01:01 PM, Nick Coghlan wrote: >PEP 432 is also related, as it includes the "pysystem" proposal [1] >(an alternate Python CLI that will default to -Es behaviour, but is >otherwise similar to the standard "python" interpreter). I *knew* this was being specified somewhere, but I couldn't find it in either the tracker or PEP summary. As an aside Nick, what do you think about splitting the pysystem proposal out of PEP 432? I think they could certainly live as independent PEPs albeit perhaps the pysystem one dependent on 432. >The rest of the discussion though makes me think we may actually need >a *nix equivalent of PEP 397 (which describes the "py" launcher we >created to get around the limitations of Windows file associations). Perhaps! >Between that and the interpreter identification mechanism defined for >the PEP 425 compatibility tags it should be possible to come up with >an upstream solution for 3.4 that the distros can backport to work >with earlier versions (similar to the way users can download the >Windows launcher directly from >https://bitbucket.org/pypa/pylauncher/downloads even though we only >started shipping it upstream as part of the Python 3.3 installer) We're getting pretty close to a real idea here. :) -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, 29 May 2013 20:10:44 +0200, Antoine Pitrou wrote: > On Wed, 29 May 2013 12:55:01 -0400 > Brett Cannon wrote: > > > Perhaps 'managed_module'? > > > > managed_module is better than managed_initialization. > > I don't understand how it's "managed". "manage", "manager", etc. is the > kind of dumb words everybody uses when they don't manage (!) to explain > what they're talking about. > > My vote is for "module_to_init", "uninitialized_module", > "pristine_module", etc. I don't really have a horse in this race (that is, whatever is chosen, my vote will be 0 on it unless someone comes up with something brilliant :), but I'll just point out that those names do not give any clue as to why the thing is a context manager instead of a function that just returns the uninitialized module. --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, 29 May 2013 20:10:44 +0200, Antoine Pitrou wrote: > On Wed, 29 May 2013 12:55:01 -0400 > Brett Cannon wrote: > > > Perhaps 'managed_module'? > > > > managed_module is better than managed_initialization. > > I don't understand how it's "managed". "manage", "manager", etc. is the > kind of dumb words everybody uses when they don't manage (!) to explain > what they're talking about. > > My vote is for "module_to_init", "uninitialized_module", > "pristine_module", etc. Actually, you are right, 'managed_module' isn't much if any better than those. Our problem is that there are two concepts we are trying to cram into one name: what the context manager is managing, and the object that the context manager gives you on entry to the with block. There probably isn't a good answer. I suppose that one approach would be to have a module_initializer context manager return self and then separately call a method on it it to actually load the module inside the with body. But adding more typing to solve a naming issue seems...odd. --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] performance testing recommendations in devguide
> Date: Wed, 29 May 2013 12:00:44 -0600 > From: ericsnowcurren...@gmail.com > To: python-dev@python.org > Subject: [Python-Dev] performance testing recommendations in devguide > > The devguide doesn't have anything on performance testing that I could > find. We do have a number of relatively useful resources in this > space though, like pybench and (eventually) speed.python.org. I'd > like to add a page to the devguide on performance testing, including > an explanation of our performance goals, how to test for them, and > what tools are available. Thanks Eric! I was looking for that kind of place! ;) > Tools I'm aware of: > * pybench (relatively limited in real-world usefulness) > * timeit module (for quick comparisions) > * benchmarks repo (real-world performance test suite) > * speed.python.org (would omit for now) Why PyBench isn't considered reliable[1]? What do you mean by "benchmarks repo"? http://hg.python.org/benchmarks ? > Things to test: > * speed > * memory (tools? tests?) > > Critically sensitive performance subjects > * interpreter start-up time > * module import overhead > * attribute lookup overhead (including MRO traversal) > * function call overhead > * instance creation overhead > * dict performance (the underlying namespace type) > * tuple performance (packing/unpacking, integral container type) > * string performance > > What would be important to say in the devguide regarding Python > performance and testing it? I've just discovered insertion at the end is faster than at the start of a list. I'd like to see things like that not only in the devguide but also in the docs (http://docs.python.org/). I found it on Dan's presentation[2] but I'm not sure it isn't in the docs somewhere. > What would you add/subtract from the > above? Threading performance! > How important is testing memory performance? How do we avoid > performance regressions? Thanks! Testing and making it faster! ;) Offcourse we need a baseline (benchmarks database) to compare and check improvements. > -eric [1] "pybench - run the standard Python PyBench benchmark suite. This is considered an unreliable, unrepresentative benchmark; do not base decisions off it. It is included only for completeness." Source: http://hg.python.org/benchmarks/file/dccd52b95a71/README.txt [2] http://stromberg.dnsalias.org/~dstromberg/Intro-to-Python/Intro%20to%20Python%202010.pdf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] performance testing recommendations in devguide
Hi, On Wed, 29 May 2013 12:00:44 -0600 Eric Snow wrote: > The devguide doesn't have anything on performance testing that I could > find. See http://bugs.python.org/issue17449 > Tools I'm aware of: > * pybench (relatively limited in real-world usefulness) > * timeit module (for quick comparisions) > * benchmarks repo (real-world performance test suite) > * speed.python.org (would omit for now) > > Things to test: > * speed > * memory (tools? tests?) You can use the "-m" option to perf.py. > Critically sensitive performance subjects > * interpreter start-up time There are startup tests in the benchmark suite. > * module import overhead > * attribute lookup overhead (including MRO traversal) > * function call overhead > * instance creation overhead > * dict performance (the underlying namespace type) > * tuple performance (packing/unpacking, integral container type) > * string performance These are all micro-benchmark fodder rather than high-level concerns (e.g. "startup time" is a high-level concern potentially impacted by "module import overhead", but only if the latter is a significant contributor to startup time). > How do we avoid performance regressions? Right now we don't have any automated way to detect them. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] performance testing recommendations in devguide
Hi, On Wed, 29 May 2013 21:59:21 +0300 Carlos Nepomuceno wrote: > > [1] "pybench - run the standard Python PyBench benchmark suite. This is > considered > an unreliable, unrepresentative benchmark; do not base decisions > off it. It is included only for completeness." "unrepresentative" is the main criticism against pybench. PyBench is a suite of micro-benchmarks (almost nano-benchmarks, actually :-)) that don't try to simulate any real-world situation. PyBench may also be unreliable, because its tests are so static that they could be optimized away by a clever enough (JIT) compiler. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] performance testing recommendations in devguide
On Wed, May 29, 2013 at 9:19 PM, Antoine Pitrou wrote: > > Hi, > > On Wed, 29 May 2013 21:59:21 +0300 > Carlos Nepomuceno wrote: >> >> [1] "pybench - run the standard Python PyBench benchmark suite. This is >> considered >> an unreliable, unrepresentative benchmark; do not base decisions >> off it. It is included only for completeness." > > "unrepresentative" is the main criticism against pybench. PyBench is a > suite of micro-benchmarks (almost nano-benchmarks, actually :-)) that > don't try to simulate any real-world situation. > > PyBench may also be unreliable, because its tests are so static that > they could be optimized away by a clever enough (JIT) compiler. > > Regards > > Antoine. For what is worth PyBench is bad because it's micro-only. A lot of stuff only shows up in larger examples, especially on an optimizing compiler. The proposed list contains also only micro-benchmarks, which will have the exact same problem as pybench. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, May 29, 2013 at 2:56 PM, R. David Murray wrote: > On Wed, 29 May 2013 20:10:44 +0200, Antoine Pitrou > wrote: >> On Wed, 29 May 2013 12:55:01 -0400 >> Brett Cannon wrote: >> > > Perhaps 'managed_module'? >> > >> > managed_module is better than managed_initialization. >> >> I don't understand how it's "managed". "manage", "manager", etc. is the >> kind of dumb words everybody uses when they don't manage (!) to explain >> what they're talking about. >> >> My vote is for "module_to_init", "uninitialized_module", >> "pristine_module", etc. I don't like unititionalized_module or pristine_module as that isn't guaranteed thanks to reloading; seems misleading. > > Actually, you are right, 'managed_module' isn't much if any better > than those. > > Our problem is that there are two concepts we are trying to cram into > one name: what the context manager is managing, and the object that the > context manager gives you on entry to the with block. There probably > isn't a good answer. > > I suppose that one approach would be to have a module_initializer context > manager return self and then separately call a method on it it to actually > load the module inside the with body. But adding more typing to solve > a naming issue seems...odd. That would make me feel icky, so I won't do it. So module_to_init it is unless someone can convince me the bikeshed is a different colour. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] performance testing recommendations in devguide
On 29.05.2013 21:19, Antoine Pitrou wrote: > > Hi, > > On Wed, 29 May 2013 21:59:21 +0300 > Carlos Nepomuceno wrote: >> >> [1] "pybench - run the standard Python PyBench benchmark suite. This is >> considered >> an unreliable, unrepresentative benchmark; do not base decisions >> off it. It is included only for completeness." > > "unrepresentative" is the main criticism against pybench. PyBench is a > suite of micro-benchmarks (almost nano-benchmarks, actually :-)) that > don't try to simulate any real-world situation. > > PyBench may also be unreliable, because its tests are so static that > they could be optimized away by a clever enough (JIT) compiler. Correct. pybench was written to test and verify CPython interpreter optimizations and also to detect changes which resulted in performance degradation of very basic operations such as attribute lookups, method calls, simple integer math, etc. It was never meant to be representative of anything :-) At the time, we only had pystone as "benchmark" and things like high precision timers were not yet readily available as they are now. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 29 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2013-07-01: EuroPython 2013, Florence, Italy ... 33 days to go : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] performance testing recommendations in devguide
29.05.13 21:00, Eric Snow написав(ла): Critically sensitive performance subjects * interpreter start-up time * module import overhead * attribute lookup overhead (including MRO traversal) * function call overhead * instance creation overhead * dict performance (the underlying namespace type) * tuple performance (packing/unpacking, integral container type) * string performance * regular expressions performance * IO performance ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, May 29, 2013 at 10:49 AM, R. David Murray wrote: > Perhaps 'managed_module'? I was just thinking the same thing. -eric ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Introduce importlib.util.ModuleManager which is a context manager to
On Wed, May 29, 2013 at 2:22 PM, Brett Cannon wrote: > So module_to_init it is unless someone can convince me the bikeshed is > a different colour. Whatever the name is, it should reflect what is happening during the with statement, and more particularly that the thing will end at the end of the with statement. managed_module() seems fine to me though it could still imply the lifetime of the module rather than the management. During the with statement the module is managed, and I expect it's clear that the management is relative to the import system. However, it could also make sense to split the function into two pieces: getting the module and handling it properly in the face of exceptions in a with statement. So, importlib.util.get_module() and ModuleType.managed(): class Loader: def load_module(self, fullname): module = importlib.util.get_module(fullname) with module.managed(): # Load/initialize the module return module If ModuleType.managed() returned the module, you could do it on one line: class Loader: def load_module(self, fullname): with importlib.util.get_module(fullname).managed() as module: # Load/initialize the module return module On second thought, that "one-liner" is a little too busy. And if it's a problem as a method on ModuleType, make it importlib.util.managed_module(): class Loader: def load_module(self, fullname): module = importlib.util.get_module(fullname) with importlib.util.managed_module(module): # Load/initialize the module return module It would be nice to have both parts in one function. It would be less boilerplate for the "common" case that way, is easier to read, and eliminates the risk of someone not realizing they need both parts. However, I'm not sure it buys us that much, the separate-part approach helps keep the two concepts distinct (for better or for worse), and each piece could be separately useful. Maybe have a third function that wraps the other two or have managed_module() accept strings (and then call get_module() internally). -eric ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
