Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Sun, Aug 30, 2009 at 5:24 PM, Guido van Rossum wrote: > On Sun, Aug 30, 2009 at 4:28 PM, Brett Cannon wrote: > > I am going through and running the entire test suite using importlib > > to ferret out incompatibilities. I have found a bunch, although all > > rather minor (raising a different exception typically; not even sure > > they are worth backporting as anyone reliant on the old exceptions > > might get a nasty surprise in the next micro release), and now I am > > down to my last failing test suite: test_import. > > > > Ignoring the execution bit problem (http://bugs.python.org/issue6526 > > but I have no clue why this is happening), I am bumping up against > > TestPycRewriting.test_incorrect_code_name. Turns out that import > > resets co_filename on a code object to __file__ before exec'ing it to > > create a module's namespace in order to ignore the file name passed > > into compile() for the filename argument. Now I can't change > > co_filename from Python as it's a read-only attribute and thus can't > > match this functionality in importlib w/o creating some custom code to > > allow me to specify the co_filename somewhere (marshal.loads() or some > > new function). > > > > My question is how important is this functionality? Do I really need > > to go through and add an argument to marshal.loads or some new > > function just to set co_filename to something that someone explicitly > > set in a .pyc file? Or I can let this go and have this be the one > > place where builtins.__import__ and importlib.__import__ differ and > > just not worry about it? > > ISTR that Bill Janssen once mentioned a file replication mechanism > whereby there were two names for each file: the "canonical" name on a > replicated read-only filesystem, and the longer "writable" name on a > unique master copy. He ended up with the filenames in the .pyc files > being pretty bogus (since not everyone had access to the writable > filesystem). So setting co_filename to match __file__ (i.e. the name > under which the module is being imported) would be a nice service in > this case. > > In general this would happen whenever you pre-compile a bunch of .py > files to .pyc/.pyo and then copy the lot to a different location. Not > a completely unlikely scenario. 8-9 years ago while using py2exe on windows to create stand along binaries out of Python programs for distribution we ran into this issue... The compiled .pyc's that py2exe bundles up contained the pathname to the source code on the development build system. When you get a stacktrace python would look for the source code based on those Really horrible if your build system used a windows drive letter other than C such as D: as it could cause windows to pop up a dialog asking the user to insert a CD or spin up a spun down optical disc or ask for a floppy, etc. ;) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fast Implementation for ZIP decryption
On Mon, Aug 31, 2009 at 12:08 PM, Gregory P. Smith wrote: > > > On Sun, Aug 30, 2009 at 10:40 PM, Jeroen Ruigrok van der Werven < > asmo...@in-nomine.org> wrote: > >> -On [20090831 06:29], Collin Winter (coll...@gmail.com) wrote: >> >Are there any applications/frameworks which have zip files on their >> >critical path, where this kind of (admittedly impressive) speedup >> >would be beneficial? What was the motivation for writing the C >> >version? >> >> Would zipped eggs count? For example, SQLAlchemy runs in the 5 MB range. >> > > Unless someone's also pushing for being able to import and execute code > from scrambled zip files, no that doesn't matter > For those who have not seen it : http://bugs.python.org/issue6749 asks for such an ability (there was a good deal of discussion about it on python-dev too and I think Greg you were a -1 on it :). > . > > The C code for this should be trivially tiny. See the > zipfile._ZipDecryptor class, its got ~25 lines of actual code in it. > right you are. It is just a simple translation of the (~25 lines) of code into C. It is not worth arguing about. I'll commit this if you post it as a patch > in a tracker issue. Please make sure your patch includes the following: > > * A unittest that compares the C version of the descrambler to the python > version of the descrambler using a variety of inputs and outputs that > exercise any boundary condition. > > * Conditional import code in the zipfile module itself so that the module > works even if the C module isn't available. > I sure can do that. What boundary conditions do you have on mind? While we are at it (and forgive my obsession with the zip module :), is there enough need for supporting the Strong Encryption Specification in the zip module? At least one immediate benefit I can see is that the OP of the link I posted above will be happy :) The main reason the idea of supporting import of encrypted module was shot down is that the simple encryption scheme is too weak to bother about. Supporting Strong Encryption might do away with that problem beside, possibly, adding a whole new way of distributing python modules. Are there any (more?) use cases or am I missing something very trivial why Strong Encryption was never supported in the zip module? -- Shashank -- Regards Shashank Singh Senior Undergraduate, Department of Computer Science and Engineering Indian Institute of Technology Bombay shashank.sunny.si...@gmail.com http://www.cse.iitb.ac.in/~shashanksingh ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is s et to?
Brett Cannon python.org> writes: > > Now I can't change > co_filename from Python as it's a read-only attribute and thus can't > match this functionality in importlib w/o creating some custom code to > allow me to specify the co_filename somewhere Why can't we simply make co_filename a writable attribute instead of inventing some complicated API? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] runpy.py
Nick Coghlan wrote: The PEPs don't go into the process of how we actually hook the command line up to the runpy module though - that's something you need to dig into the main.c code to really understand. Yeah, main.c does quite a lot... ;-) This all spawned from a suggestion by Jim Fulton over on the distutils-sig that it would be nice if there was a python module that did all of the various types of launching found in main.c. His use case is so that buildout scripts can easily use the same functionality that the interpreter startup uses. I didn't spot any, but does anyone know of code in that mix that couldn't be moved to a pure python module like runpy? If not, how would people feel about the various types of launching all moving to runpy rather than just the -m stuff being there? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Excluding the current path from module search path?
Nick Coghlan wrote: Ah, OK - I see the problem now. However, I think the current behaviour is correct, it just needs to be documented better (probably noted in both the command line doco Not sure what you mean by this? regarding sys.path manipulation and in the doco for site.py). Agreed :-) The reason I think the current behaviour is correct is that site.py and sitecustomize.py are meant to be about customising the *site* (i.e. the installation of Python that is being executed) rather than about customizing a particular application. Unless you use virtualenv as Guido suggested in the other thread ;-) Importing them before the script specific directories are prepended to sys.path goes a long way towards achieving that. If sitecustomize.py had more uses that the setdefaultencoding hack, I'd argue more about this... If it does have other uses, my argument would be that "site" wide is a very subjective term tht many people, myself included, would like to be able to mean "per project, I don't *ever* want to screw with my actual Python install, it should stay pristine"... Also, as was pointed out on the tracker item, having a script that can automatically be executed when running an arbitrary Python script without any request from or notification to the user is not a good idea from a security standpoint. Agreed, but I think that's only an issue when you're starting up an interpreter. If you're running a script from a file or module, I'd say it's more akin to what's specified in PYTHONSTARTUP being executed that than a random script being silently executed without your permission. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] deleting setdefaultencoding iin site.py is evil
Guido van Rossum wrote: Being adults about it also means when to give up. Chris, please stop arguing about this. Sure. Even if people had agreed to this change, it wouldn't end up in a python release I could use for this project. There are plenty of techniques you can use to get what you want without changing Python, for example virtualenv, which allows you to create a custom Python environment for each project. Yep, I'll resort to wrapping the buildout in a virtualenv iff the reload(sys) hack ends up causing problems... Or you could switch to Python 3.1, I would love to, once Python 3 has a viable web app story... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
Brett Cannon wrote: >> You can use the C implementation of io, _io, which has a full >> buffering implementation. Of course, that also makes it a better >> harder for other implementations which may wish to use importlib >> because the io library would have to be completely implemented... > > True. I guess it's a question of whether making importlib easier to > maintain and as minimally reliant on C-specific modules is more/less > important than trying to bootstrap it in for CPython for __import__ at > some point. I'd suggest preferring _io, but falling back to the Python io module if the accelerated version doesn't exist. You should get the best of both worlds that way (no bootstrap issues in CPython and other implementations with an _io module, but a still functional importlib in other implementations). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
Antoine Pitrou wrote: > Brett Cannon python.org> writes: >> Now I can't change >> co_filename from Python as it's a read-only attribute and thus can't >> match this functionality in importlib w/o creating some custom code to >> allow me to specify the co_filename somewhere > > Why can't we simply make co_filename a writable attribute instead of inventing > some complicated API? I thought of that question as well, but the later exchange between Guido and Brett made me realise that a lot more than the top level module code object is affected here - the adjustment also needs to be propagated to the code objects created by the module for functions and generators and so forth. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
Nick Coghlan wrote: Antoine Pitrou wrote: Brett Cannon python.org> writes: Now I can't change co_filename from Python as it's a read-only attribute and thus can't match this functionality in importlib w/o creating some custom code to allow me to specify the co_filename somewhere Why can't we simply make co_filename a writable attribute instead of inventing some complicated API? I thought of that question as well, but the later exchange between Guido and Brett made me realise that a lot more than the top level module code object is affected here - the adjustment also needs to be propagated to the code objects created by the module for functions and generators and so forth. Even if it is not necessary or sufficient it still sounds like a useful change. When writing tools that generate modules or manipulate code objects these read-only attributes are a great nuisance. Michael Cheers, Nick. -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
Nick Coghlan gmail.com> writes: > > I thought of that question as well, but the later exchange between Guido > and Brett made me realise that a lot more than the top level module code > object is affected here - the adjustment also needs to be propagated to > the code objects created by the module for functions and generators and > so forth. I'm not sure I understand. There's a single type of "code object" and it's PyCodeObject. Making the attribute writable (from Python code) at that level should be sufficient. (then of course the recursive machinery needed to mutate all code objects created in a module may be slightly inefficient if written in Python, but at least it's possible to write it) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] runpy.py
Chris Withers wrote: > Nick Coghlan wrote: >> The PEPs don't go into the process of how we actually hook the command >> line up to the runpy module though - that's something you need to dig >> into the main.c code to really understand. > > Yeah, main.c does quite a lot... ;-) > > This all spawned from a suggestion by Jim Fulton over on the > distutils-sig that it would be nice if there was a python module that > did all of the various types of launching found in main.c. His use case > is so that buildout scripts can easily use the same functionality that > the interpreter startup uses. > > I didn't spot any, but does anyone know of code in that mix that > couldn't be moved to a pure python module like runpy? > > If not, how would people feel about the various types of launching all > moving to runpy rather than just the -m stuff being there? I haven't timed it, but I believe runpy is a fair bit slower than the native C functions in main. (That first part of the comment means I could easily be wrong though - it's definitely possible that overall interpreter startup time will dwarf any difference between the two launch mechanisms). That said, while actually ditching the C code might cause an argument, expanding runpy with Python equivalents of the C level functionality (i.e. run script by name, run directory/zipfile by name, '-c' switch, and other odds and ends that I'm probably forgetting right now, with all associated modifications to sys.argv and the __main__ module attributes) should be far less controversial. For example, _run_module_as_main() has survived long enough now without anyone poking holes in it (unlike the holes in the original run_module() that PJE drove a truck through!) that I could probably be talked into removing the comment I put on it and making it public :) As you say, making all of that functionality accessible from Python would allow launch scripts to be far more flexible in handling arguments as if they were the normal interpreter. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
2009/8/31 Antoine Pitrou : > Brett Cannon python.org> writes: >> >> Now I can't change >> co_filename from Python as it's a read-only attribute and thus can't >> match this functionality in importlib w/o creating some custom code to >> allow me to specify the co_filename somewhere > > Why can't we simply make co_filename a writable attribute instead of inventing > some complicated API? Because code objects are supposed to be a immutable hashable object? -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] deleting setdefaultencoding iin site.py is evil
Chris Withers wrote: > Guido van Rossum wrote: > > Being adults about it also means when to give up. Chris, please stop > > arguing about this. > > Sure. Even if people had agreed to this change, it wouldn't end up in a > python release I could use for this project. > > > There are plenty of techniques you can use to get > > what you want without changing Python, for example virtualenv, which > > allows you to create a custom Python environment for each project. > > Yep, I'll resort to wrapping the buildout in a virtualenv iff the > reload(sys) hack ends up causing problems... > > > Or > > you could switch to Python 3.1, > > I would love to, once Python 3 has a viable web app story... CherryPy 3.2 is now in beta, and mod_wsgi is nearly ready as well. Both support Python 3. :) Robert Brewer fuman...@aminus.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] web apps in python 3
Robert Brewer wrote: you could switch to Python 3.1, I would love to, once Python 3 has a viable web app story... CherryPy 3.2 is now in beta, and mod_wsgi is nearly ready as well. Both support Python 3. :) My understanding was that the wsgi spec for Python 3 wasn't finished... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is s et to?
Benjamin Peterson python.org> writes: > > > Why can't we simply make co_filename a writable attribute instead of inventing > > some complicated API? > > Because code objects are supposed to be a immutable hashable object? Right, but co_filename is used neither in tp_hash nor in tp_richcompare. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] web apps in python 3
Chris Withers wrote: > Robert Brewer wrote: you could switch to Python 3.1, >>> I would love to, once Python 3 has a viable web app story... >> >> CherryPy 3.2 is now in beta, and mod_wsgi is nearly ready as well. Both >> support Python 3. :) > > My understanding was that the wsgi spec for Python 3 wasn't finished... The WSGI 1.0 spec has always included Python 3 using unicode strings in the environ (decoded via ISO-8859-1, and limited to \x00-\xFF). In addition, the CherryPy and mod_wsgi teams are working to interoperably support a modified version of WSGI, in which the environ is true unicode for both Python 2 and 3. We hope these implementations become references from which a WSGI 1.1 spec can be written; since web-sig has not yet reached consensus on certain specification details, we are proceeding together with tools that allow users to get work done now. Robert Brewer fuman...@aminus.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] deleting setdefaultencoding iin site.py is evil
On Mon, Aug 31, 2009 at 7:49 AM, Robert Brewer wrote: > CherryPy 3.2 is now in beta, and mod_wsgi is nearly ready as well. Both > support Python 3. :) Excellent news! I just saw that PyYAML also suports 3.1. Slowly but surely, 3.1 is gaining traction... -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Mon, Aug 31, 2009 at 08:10, Antoine Pitrou wrote: > Benjamin Peterson python.org> writes: >> >> > Why can't we simply make co_filename a writable attribute instead of > inventing >> > some complicated API? >> >> Because code objects are supposed to be a immutable hashable object? > > Right, but co_filename is used neither in tp_hash nor in tp_richcompare. I didn't suggest this since I assumed co_filename was made read-only for a reason back when the design decision was made. But if the original safety concerns are not there then I am happy to simply change the attribute to writable. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Mon, Aug 31, 2009 at 9:27 AM, Brett Cannon wrote: > On Mon, Aug 31, 2009 at 08:10, Antoine Pitrou wrote: >> Benjamin Peterson python.org> writes: >>> >>> > Why can't we simply make co_filename a writable attribute instead of >> inventing >>> > some complicated API? >>> >>> Because code objects are supposed to be a immutable hashable object? >> >> Right, but co_filename is used neither in tp_hash nor in tp_richcompare. > > I didn't suggest this since I assumed co_filename was made read-only > for a reason back when the design decision was made. But if the > original safety concerns are not there then I am happy to simply > change the attribute to writable. Hm... I still wonder if there would be bad side effects of making co_filename writable, but I can't think of any, so maybe you can make this work... The next step would be to not write it out when marshalling a code object -- this might save a bit of space in pyc files too! (I guess for compatibility you might want to write it as an empty string.) Of course, tracking down all the code objects in the return value of marshal.load*() might be a bit tricky -- API-wise I still think that making it an argument to marshal.load*() might be simpler. Also it would preserve the purity of code objects. (Michael: it would be fine if *other* implementations of Python made co_filename writable, as long as you can't think of security issues with this.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Mon, Aug 31, 2009 at 06:13, Nick Coghlan wrote: > Brett Cannon wrote: >>> You can use the C implementation of io, _io, which has a full >>> buffering implementation. Of course, that also makes it a better >>> harder for other implementations which may wish to use importlib >>> because the io library would have to be completely implemented... >> >> True. I guess it's a question of whether making importlib easier to >> maintain and as minimally reliant on C-specific modules is more/less >> important than trying to bootstrap it in for CPython for __import__ at >> some point. > > I'd suggest preferring _io, but falling back to the Python io module if > the accelerated version doesn't exist. You should get the best of both > worlds that way (no bootstrap issues in CPython and other > implementations with an _io module, but a still functional importlib in > other implementations). Well, all important code is in importlib._bootstrap which lacks a single import statement; all dependent modules are injected externally in importlib.__init__. That allows for the possibility of C code to import importlib/_bootstrap.py along with the buit-in modules required, and then inject those built-in modules into importlib's global namespace. This is why I have functions in there that are duplications of things found elsewhere. That means that while I have named the module _io and I use _io.FileIO, you could also easily inject io with the name _io and have everything just work if you were trying to bootstrap. The deal is that if I want to keep up the bootstrap goal I need to continue to restrict myself to either built-in modules or thing I know I can choose to expose later in built-in modules. This is why when Guido suggested os.open I said I will have to see how it is implemented because if it doesn't come from posix or is easy to duplicate I won't be able to use it. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Mon, Aug 31, 2009 at 09:33, Guido van Rossum wrote: > On Mon, Aug 31, 2009 at 9:27 AM, Brett Cannon wrote: >> On Mon, Aug 31, 2009 at 08:10, Antoine Pitrou wrote: >>> Benjamin Peterson python.org> writes: > Why can't we simply make co_filename a writable attribute instead of >>> inventing > some complicated API? Because code objects are supposed to be a immutable hashable object? >>> >>> Right, but co_filename is used neither in tp_hash nor in tp_richcompare. >> >> I didn't suggest this since I assumed co_filename was made read-only >> for a reason back when the design decision was made. But if the >> original safety concerns are not there then I am happy to simply >> change the attribute to writable. > > Hm... I still wonder if there would be bad side effects of making > co_filename writable, but I can't think of any, so maybe you can make > this work... The next step would be to not write it out when > marshalling a code object -- this might save a bit of space in pyc > files too! (I guess for compatibility you might want to write it as an > empty string.) I would only want to consider stripping out the filename from the marshal format if a filename argument to marshal.load* was required to guarantee that code objects always in some sensible state. Otherwise everyone would end up with tracebacks that made no sense by default. But adding a required argument to marshal.load* would be quite the pain for compatibility. > > Of course, tracking down all the code objects in the return value of > marshal.load*() might be a bit tricky -- API-wise I still think that > making it an argument to marshal.load*() might be simpler. Also it > would preserve the purity of code objects. > > (Michael: it would be fine if *other* implementations of Python made > co_filename writable, as long as you can't think of security issues > with this.) OK, so what does co_filename get used for? I think it is referenced to open files for use in printing out the traceback. Python won't be able to open files that you can't as a user, so that shouldn't be a security risk. All places where co_filename is referenced would need to gain a check or start using some new C function/macro which verified that co_filename was a string and not some number or something else which wouldn't get null-terminated and thus lead to buffer overflow. A quick grep for co_filename turns up 17 uses in C code, although having to add some check would ruin the purity Guido is talking about and make a single attribute on code objects something people have to be careful about instead of having a guarantee that all attributes have some specific type of value. I'm with Guido; I would rather add an optional argument to marshal.load*. It must be a string and, if present, is used to override co_filename in the resulting code object. Once we have had the argument around we can then potentially make it a required argument and have file paths in the marshal data go away (or decide to default to some string constant when people don't specify the path argument). -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Mon, Aug 31, 2009 at 9:57 AM, Brett Cannon wrote: > On Mon, Aug 31, 2009 at 09:33, Guido van Rossum wrote: >> Hm... I still wonder if there would be bad side effects of making >> co_filename writable, but I can't think of any, so maybe you can make >> this work... The next step would be to not write it out when >> marshalling a code object -- this might save a bit of space in pyc >> files too! (I guess for compatibility you might want to write it as an >> empty string.) > > I would only want to consider stripping out the filename from the > marshal format if a filename argument to marshal.load* was required to > guarantee that code objects always in some sensible state. Otherwise > everyone would end up with tracebacks that made no sense by default. > But adding a required argument to marshal.load* would be quite the > pain for compatibility. Well... It would be, but consider this: marshal.load() already takes a file argument; in most cases you can extract the name from the file easily. And for marshal.loads(), I'm not sure that the filename baked into the data is all that reliable anyways. >> Of course, tracking down all the code objects in the return value of >> marshal.load*() might be a bit tricky -- API-wise I still think that >> making it an argument to marshal.load*() might be simpler. Also it >> would preserve the purity of code objects. >> >> (Michael: it would be fine if *other* implementations of Python made >> co_filename writable, as long as you can't think of security issues >> with this.) > > OK, so what does co_filename get used for? I think it is referenced to > open files for use in printing out the traceback. Python won't be able > to open files that you can't as a user, so that shouldn't be a > security risk. All places where co_filename is referenced would need > to gain a check or start using some new C function/macro which > verified that co_filename was a string and not some number or > something else which wouldn't get null-terminated and thus lead to > buffer overflow. You could also do the validation on assignment. > A quick grep for co_filename turns up 17 uses in C > code, although having to add some check would ruin the purity Guido is > talking about and make a single attribute on code objects something > people have to be careful about instead of having a guarantee that all > attributes have some specific type of value. > > I'm with Guido; I would rather add an optional argument to > marshal.load*. It must be a string and, if present, is used to > override co_filename in the resulting code object. Once we have had > the argument around we can then potentially make it a required > argument and have file paths in the marshal data go away (or decide to > default to some string constant when people don't specify the path > argument). Actually that sounds like a fine transitional argument. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Mon, Aug 31, 2009 at 10:02, Guido van Rossum wrote: > On Mon, Aug 31, 2009 at 9:57 AM, Brett Cannon wrote: >> On Mon, Aug 31, 2009 at 09:33, Guido van Rossum wrote: >>> Hm... I still wonder if there would be bad side effects of making >>> co_filename writable, but I can't think of any, so maybe you can make >>> this work... The next step would be to not write it out when >>> marshalling a code object -- this might save a bit of space in pyc >>> files too! (I guess for compatibility you might want to write it as an >>> empty string.) >> >> I would only want to consider stripping out the filename from the >> marshal format if a filename argument to marshal.load* was required to >> guarantee that code objects always in some sensible state. Otherwise >> everyone would end up with tracebacks that made no sense by default. >> But adding a required argument to marshal.load* would be quite the >> pain for compatibility. > > Well... It would be, but consider this: marshal.load() already takes a > file argument; in most cases you can extract the name from the file > easily. And for marshal.loads(), I'm not sure that the filename baked > into the data is all that reliable anyways. > >>> Of course, tracking down all the code objects in the return value of >>> marshal.load*() might be a bit tricky -- API-wise I still think that >>> making it an argument to marshal.load*() might be simpler. Also it >>> would preserve the purity of code objects. >>> >>> (Michael: it would be fine if *other* implementations of Python made >>> co_filename writable, as long as you can't think of security issues >>> with this.) >> >> OK, so what does co_filename get used for? I think it is referenced to >> open files for use in printing out the traceback. Python won't be able >> to open files that you can't as a user, so that shouldn't be a >> security risk. All places where co_filename is referenced would need >> to gain a check or start using some new C function/macro which >> verified that co_filename was a string and not some number or >> something else which wouldn't get null-terminated and thus lead to >> buffer overflow. > > You could also do the validation on assignment. > >> A quick grep for co_filename turns up 17 uses in C >> code, although having to add some check would ruin the purity Guido is >> talking about and make a single attribute on code objects something >> people have to be careful about instead of having a guarantee that all >> attributes have some specific type of value. >> >> I'm with Guido; I would rather add an optional argument to >> marshal.load*. It must be a string and, if present, is used to >> override co_filename in the resulting code object. Once we have had >> the argument around we can then potentially make it a required >> argument and have file paths in the marshal data go away (or decide to >> default to some string constant when people don't specify the path >> argument). > > Actually that sounds like a fine transitional argument. I will plan to take this approach then; http://bugs.python.org/issue6811 will track all of this. Since this is a 3.2 thing I am not going to rush to implement this. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
At 09:33 AM 8/31/2009 -0700, Guido van Rossum wrote: Of course, tracking down all the code objects in the return value of marshal.load*() might be a bit tricky -- API-wise I still think that making it an argument to marshal.load*() might be simpler. Also it would preserve the purity of code objects. Or maybe we could just do something like this: from new import code def with_changed_filename(code_ob, filename): def remap(ob): if not isinstance(ob, code): return ob return code( ob.co_argcount, ob.co_nlocals, ob.co_stacksize, ob.co_flags, ob.co_code, map(remap, ob.co_consts), ob.co_names, ob.co_varnames, filename, ob.co_name, ob.co_firstlineno, ob.co_lnotab, ob.co_freevars, ob.co_cellvars ) return remap(code_ob) Granted, this takes a bit more memory than an in-place modification, but it's immediately usable and at least works wherever new.code is available. (I've not tested the above, so it may not work. I seem to recall the last time I wrote something like this there was something tricky about handling co_freevars and co_cellvars; I think you may need to omit them if empty, or convert them to None, or from None to an empty tuple or some such rigamarole. And a 3.x version is left as an exercise for the reader. ;-) ) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3to2 0.1 alpha 1 released
On Wed, 26 Aug 2009 15:55:54 -0700, Joe Amenta wrote: -- 3to2 is now registered with PyPI. Did I do it right? http://pypi.python.org/pypi/3to2/0.1%20alpha%201 Please fix the version number to not contain any whitespace characters. Also set the `version` argument in setup(..) in your setup.py. And then you may want to use the `upload` command to upload the new tarball to PyPI. See http://wiki.python.org/moin/Distutils/Tutorial for more details. -srid ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] runpy.py
On Mon, Aug 31, 2009 at 06:36, Nick Coghlan wrote: > Chris Withers wrote: >> Nick Coghlan wrote: >>> The PEPs don't go into the process of how we actually hook the command >>> line up to the runpy module though - that's something you need to dig >>> into the main.c code to really understand. >> >> Yeah, main.c does quite a lot... ;-) >> >> This all spawned from a suggestion by Jim Fulton over on the >> distutils-sig that it would be nice if there was a python module that >> did all of the various types of launching found in main.c. His use case >> is so that buildout scripts can easily use the same functionality that >> the interpreter startup uses. >> >> I didn't spot any, but does anyone know of code in that mix that >> couldn't be moved to a pure python module like runpy? >> >> If not, how would people feel about the various types of launching all >> moving to runpy rather than just the -m stuff being there? > > I haven't timed it, but I believe runpy is a fair bit slower than the > native C functions in main. (That first part of the comment means I > could easily be wrong though - it's definitely possible that overall > interpreter startup time will dwarf any difference between the two > launch mechanisms). > That's quite possible. If you benchmark it you might be able to convince people. > That said, while actually ditching the C code might cause an argument, > expanding runpy with Python equivalents of the C level functionality > (i.e. run script by name, run directory/zipfile by name, '-c' switch, > and other odds and ends that I'm probably forgetting right now, with all > associated modifications to sys.argv and the __main__ module attributes) > should be far less controversial. > It also has the perk of letting alternative VMs not have to implement all of that stuff themselves, potentially helping to unify even the command-line interfaces for all the VMs. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3to2 0.1 alpha 1 released
On Mon, Aug 31, 2009 at 12:18, Sridhar Ratnakumar wrote: > On Wed, 26 Aug 2009 15:55:54 -0700, Joe Amenta wrote: > >> -- 3to2 is now registered with PyPI. Did I do it right? > >> http://pypi.python.org/pypi/3to2/0.1%20alpha%201 > > Please fix the version number to not contain any whitespace characters. Also > set the `version` argument in setup(..) in your setup.py. And then you may > want to use the `upload` command to upload the new tarball to PyPI. See > http://wiki.python.org/moin/Distutils/Tutorial for more details. See PEP 386 (http://www.python.org/dev/peps/pep-0386/) for what the current thinking on version numbers is. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is s et to?
Brett Cannon python.org> writes: > > I will plan to take this approach then; > http://bugs.python.org/issue6811 will track all of this. Since this is > a 3.2 thing I am not going to rush to implement this. I still don't understand what the point is of this complicated approach (adding an argument to marshal.load()) compared to the simple and obvious approach (making co_filename mutable). Besides, the latter would let you code the recursive renaming algorithm in Python, which is the whole point of importlib (rewriting most code in Python), isn't it? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Mon, Aug 31, 2009 at 12:27, Antoine Pitrou wrote: > Brett Cannon python.org> writes: >> >> I will plan to take this approach then; >> http://bugs.python.org/issue6811 will track all of this. Since this is >> a 3.2 thing I am not going to rush to implement this. > > I still don't understand what the point is of this complicated approach > (adding > an argument to marshal.load()) compared to the simple and obvious approach > (making co_filename mutable). If we add the argument to marshal.load* we can eventually drop the file location string from marshal data entirely by requiring people to specify the filename to use when the code object is created. Making co_filename mutable simply doesn't allow for this case unless we decide a default value should be used instead. > Besides, the latter would let you code the recursive renaming algorithm in > Python, which is the whole point of importlib (rewriting most code in Python), > isn't it? Sure, but I am not about to re-implement marshal in pure Python just because importlib uses it. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Mon, Aug 31, 2009 at 12:59, Brett Cannon wrote: > On Mon, Aug 31, 2009 at 12:27, Antoine Pitrou wrote: >> Brett Cannon python.org> writes: >>> >>> I will plan to take this approach then; >>> http://bugs.python.org/issue6811 will track all of this. Since this is >>> a 3.2 thing I am not going to rush to implement this. >> >> I still don't understand what the point is of this complicated approach >> (adding >> an argument to marshal.load()) compared to the simple and obvious approach >> (making co_filename mutable). > > If we add the argument to marshal.load* we can eventually drop the > file location string from marshal data entirely by requiring people to > specify the filename to use when the code object is created. Making > co_filename mutable simply doesn't allow for this case unless we > decide a default value should be used instead. > I should also mention that I am +0 on the marshal.load* change. I could be convinced to try to pursue a mutable co_filenme direction, but considering the BDFL likes the marshal.load* approach and it opens up the possibility of compacting the marshal format I am leaning towards sticking with this initial direction. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is s et to?
Brett Cannon python.org> writes: > > I should also mention that I am +0 on the marshal.load* change. I > could be convinced to try to pursue a mutable co_filenme direction, > but considering the BDFL likes the marshal.load* approach and it opens > up the possibility of compacting the marshal format I am leaning > towards sticking with this initial direction. I am really not opinionated on this one. I was just pointing out that choosing a non-obvious solution generally requires good reasons to do so. The marshal format compaction sounds like premature optimization, since nobody seems to have formulated such a request. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] default of returning None hurts performance?
food for thought as noticed by a coworker who has been profiling some hot code to optimize a library... If a function does not have a return statement we return None. Ironically this makes the foo2 function below faster than the bar2 function at least as measured using bytecode size: Python 2.6.2 (r262:71600, Jul 24 2009, 17:29:21) [GCC 4.2.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import dis >>> def foo(x): ... y = x() ... return y ... >>> def foo2(x): ... return x() ... >>> def bar(x): ... y = x() ... >>> def bar2(x): ... x() ... >>> dis.dis(foo) 2 0 LOAD_FAST0 (x) 3 CALL_FUNCTION0 6 STORE_FAST 1 (y) 3 9 LOAD_FAST1 (y) 12 RETURN_VALUE >>> dis.dis(foo2) 2 0 LOAD_FAST0 (x) 3 CALL_FUNCTION0 6 RETURN_VALUE >>> dis.dis(bar) 2 0 LOAD_FAST0 (x) 3 CALL_FUNCTION0 6 STORE_FAST 1 (y) 9 LOAD_CONST 0 (None) 12 RETURN_VALUE >>> dis.dis(bar2) 2 0 LOAD_FAST0 (x) 3 CALL_FUNCTION0 6 POP_TOP 7 LOAD_CONST 0 (None) 10 RETURN_VALUE ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
Gregory P. Smith krypto.org> writes: > > food for thought as noticed by a coworker who has been profiling some hot code to optimize a library...If a function does not have a return statement we return None. Ironically this makes the foo2 function below faster than the bar2 function at least as measured using bytecode size I would be surprised if this "bytecode size" difference made a significant difference in runtimes, given that function call cost should dwarf the cumulated cost of POP_TOP and LOAD_CONST (two of the simplest opcodes you could find). Did your coworker run any timings instead of basing his assumptions on bytecode size? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144: IP Address Manipulation Library for the Python Standard Library
On Fri, Aug 28, 2009 at 2:31 AM, Nick Coghlan wrote: > Peter Moody wrote: >> If there are any more suggestions on the PEP or the code, please let me know. > > I noticed the new paragraphs on the IPv4 vs IPv6 types not being > comparable - is there a canonical ordering for mixed address lists > defined anywhere (e.g. an RFC)? > > If there is, then it should be possible to implement that on BaseIP and > BaseNet so that comparisons work as canonically defined. If there isn't, > then that should be mentioned in the PEP as the reason why the PEP > deliberately isn't trying to invent a convention. updated the pep with more information about this. working through changes/issues brought up by David and Martin. Cheers, /peter > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > --- > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/python-dev%40hda3.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
At 01:39 PM 8/31/2009 -0700, Brett Cannon wrote: On Mon, Aug 31, 2009 at 12:59, Brett Cannon wrote: > On Mon, Aug 31, 2009 at 12:27, Antoine Pitrou wrote: >> Brett Cannon python.org> writes: >>> >>> I will plan to take this approach then; >>> http://bugs.python.org/issue6811 will track all of this. Since this is >>> a 3.2 thing I am not going to rush to implement this. >> >> I still don't understand what the point is of this complicated approach (adding >> an argument to marshal.load()) compared to the simple and obvious approach >> (making co_filename mutable). > > If we add the argument to marshal.load* we can eventually drop the > file location string from marshal data entirely by requiring people to > specify the filename to use when the code object is created. Making > co_filename mutable simply doesn't allow for this case unless we > decide a default value should be used instead. > I should also mention that I am +0 on the marshal.load* change. I could be convinced to try to pursue a mutable co_filenme direction, but considering the BDFL likes the marshal.load* approach and it opens up the possibility of compacting the marshal format I am leaning towards sticking with this initial direction. Why not just try the code I posted earlier, that doesn't need a mutable attribute OR an API change? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Mon, Aug 31, 2009 at 14:52, P.J. Eby wrote: > At 01:39 PM 8/31/2009 -0700, Brett Cannon wrote: >> >> On Mon, Aug 31, 2009 at 12:59, Brett Cannon wrote: >> > On Mon, Aug 31, 2009 at 12:27, Antoine Pitrou >> > wrote: >> >> Brett Cannon python.org> writes: >> >>> >> >>> I will plan to take this approach then; >> >>> http://bugs.python.org/issue6811 will track all of this. Since this is >> >>> a 3.2 thing I am not going to rush to implement this. >> >> >> >> I still don't understand what the point is of this complicated approach >> >> (adding >> >> an argument to marshal.load()) compared to the simple and obvious >> >> approach >> >> (making co_filename mutable). >> > >> > If we add the argument to marshal.load* we can eventually drop the >> > file location string from marshal data entirely by requiring people to >> > specify the filename to use when the code object is created. Making >> > co_filename mutable simply doesn't allow for this case unless we >> > decide a default value should be used instead. >> > >> >> I should also mention that I am +0 on the marshal.load* change. I >> could be convinced to try to pursue a mutable co_filenme direction, >> but considering the BDFL likes the marshal.load* approach and it opens >> up the possibility of compacting the marshal format I am leaning >> towards sticking with this initial direction. > > Why not just try the code I posted earlier, that doesn't need a mutable > attribute OR an API change? Ignoring that 'new' is not in Python 3.x (luckily 'types' is), I want a proper solution that doesn't require reconstructing every code object that I happen to import. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Aug 31, 2009, at 4:47 PM, Antoine Pitrou wrote: I am really not opinionated on this one. I was just pointing out that choosing a non-obvious solution generally requires good reasons to do so. The marshal format compaction sounds like premature optimization, since nobody seems to have formulated such a request. Every time I've been bitten by the wrong co_filename values (usually from tracebacks), changing the way marshal creates code objects to use a values passed in has been the thing that made the most sense to me. The feature request that's involved here, getting correct co_filename values, can be implemented in different ways, sure. This particular change produces the least impact in the because it *doesn't* change the mutability of code objects. I for one appreciate that, mostly because I'm simply wary of making code objects mutable in this way having unexpected side effects in some library. -Fred -- Fred Drake ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
At 02:57 PM 8/31/2009 -0700, Brett Cannon wrote: Ignoring that 'new' is not in Python 3.x (luckily 'types' is), I want a proper solution that doesn't require reconstructing every code object that I happen to import. Practicality beats purity. ;-) (Especially if it allows importlib to run on older Pythons.) Also, surely you're not worried about *performance* here? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how important is setting co_filename for a module being imported to what __file__ is set to?
On Mon, Aug 31, 2009 at 15:06, P.J. Eby wrote: > At 02:57 PM 8/31/2009 -0700, Brett Cannon wrote: >> >> Ignoring that 'new' is not in Python 3.x (luckily 'types' is), I want >> a proper solution that doesn't require reconstructing every code >> object that I happen to import. > > Practicality beats purity. ;-) (Especially if it allows importlib to run > on older Pythons.) > I don't care about making importlib run on older versions of Python before 3.1. And this is a minor enough thing that I am not worried about missing in Python 3.1. > Also, surely you're not worried about *performance* here? I do care about performance to an extent, but that is not the primary motivating factor to wanting to go with the marshal API change. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
On Mon, Aug 31, 2009 at 2:20 PM, Antoine Pitrou wrote: > Gregory P. Smith krypto.org> writes: > > > > food for thought as noticed by a coworker who has been profiling some hot > code > to optimize a library...If a function does not have a return statement we > return > None. Ironically this makes the foo2 function below faster than the bar2 > function at least as measured using bytecode size > > I would be surprised if this "bytecode size" difference made a significant > difference in runtimes, given that function call cost should dwarf the > cumulated > cost of POP_TOP and LOAD_CONST (two of the simplest opcodes you could > find). > > the attached sample code repeatably shows that it makes a difference though its really not much of one (2-3%). I was just wondering if a bytecode for a superinstruction of the common sequence: 6 POP_TOP 7 LOAD_CONST 0 (None) 10 RETURN_VALUE might be worth it. #!/usr/bin/python2.4 import dis import time def returns_none(x): x() def returns_call(x): return x() def dummy(): pass print 'returns_none:' dis.dis(returns_none) start, start_clock = time.time(), time.clock() for _ in xrange(1000): returns_none(dummy) stop, stop_clock = time.time(), time.clock() print 'elapsed time', stop-start, stop_clock-start_clock print 'returns_call:' dis.dis(returns_call) start, start_clock = time.time(), time.clock() for _ in xrange(1000): returns_call(dummy) stop, stop_clock = time.time(), time.clock() print 'elapsed time', stop-start, stop_clock-start_clock ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
On Mon, Aug 31, 2009 at 3:07 PM, Gregory P. Smith wrote: > > > On Mon, Aug 31, 2009 at 2:20 PM, Antoine Pitrou wrote: >> >> Gregory P. Smith krypto.org> writes: >> > >> > food for thought as noticed by a coworker who has been profiling some >> > hot code >> to optimize a library...If a function does not have a return statement we >> return >> None. Ironically this makes the foo2 function below faster than the bar2 >> function at least as measured using bytecode size >> >> I would be surprised if this "bytecode size" difference made a significant >> difference in runtimes, given that function call cost should dwarf the >> cumulated >> cost of POP_TOP and LOAD_CONST (two of the simplest opcodes you could >> find). >> > > the attached sample code repeatably shows that it makes a difference though > its really not much of one (2-3%). > > I was just wondering if a bytecode for a superinstruction of the common > sequence: > > 6 POP_TOP > 7 LOAD_CONST 0 (None) > 10 RETURN_VALUE > > might be worth it. I doubt it. You'd save a bit of stack manipulation, but since this will only appear at the end of a function, I'd be skeptical that this would make any macrobenchmarks (statistically) significantly faster. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
2009/8/31 Gregory P. Smith : > I was just wondering if a bytecode for a superinstruction of the common > sequence: > > 6 POP_TOP > 7 LOAD_CONST 0 (None) > 10 RETURN_VALUE > > might be worth it. We could just utilize RETURN_VALUE's op argument. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
Gregory P. Smith krypto.org> writes: > > I was just wondering if a bytecode for a superinstruction of the common sequence: > 6 POP_TOP > 7 LOAD_CONST 0 (None) > 10 RETURN_VALUE > might be worth it. I think superinstructions in general would be a good thing to experiment, as wpython showed. Direct addressing (via a pseudo register file combining locals and constants) would eliminate many bookkeeping-related opcodes in common bytecode. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
I was just wondering if a bytecode for a superinstruction of the common sequence: 6 POP_TOP 7 LOAD_CONST 0 (None) 10 RETURN_VALUE might be worth it. [Collin Winter] I doubt it. You'd save a bit of stack manipulation, but since this will only appear at the end of a function, I'd be skeptical that this would make any macrobenchmarks (statistically) significantly faster. I concur with Collin. And since it appears only at the end of a function, the optimization doesn't help inner-loops in a function (where most of the time usually spent). Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
Antoine Pitrou wrote: Did your coworker run any timings instead of basing his assumptions on bytecode size? In any case, what are you suggesting -- that the last value returned by a function call in the body should be the default return value? I don't think the unpredictability that would introduce would be a good idea. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
On Mon, Aug 31, 2009 at 5:01 PM, Greg Ewing wrote: > Antoine Pitrou wrote: > > Did your coworker run any timings instead of basing his assumptions on >> bytecode >> size? >> > > In any case, what are you suggesting -- that the last value > returned by a function call in the body should be the > default return value? > > I don't think the unpredictability that would introduce > would be a good idea. > gads no. we're not shell or perl! don't do that :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] why different between staticmethod and classmethod on non-callable object?
hi all:
In the svn thunk tree, I find in file Object/funcobject.c, the two
functions: sm_init and cm_init have a little different. the cm_init
function will check the object is really callable, but the sm_init
don't.
the code like this:
if (!PyCallable_Check(callable)) {
PyErr_Format(PyExc_TypeError, "'%s' object is not callable",
callable->ob_type->tp_name);
return -1;
}
so when use staticmethod and classmethod on a class variable, there
are two different behavior.
class Test(object):
name = "s7v7nislands"
name = classmethod(Test.name) # will raise error.
name = staticmethod(Test.name) # will not raise error.
My idea is: here, the two functions (or maybe classes) should have the
same behavior).
so is this a bug or something I missing ?
thanks!
s7v7nislands
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3to2 0.1 alpha 1 released
Thanks Brett and Sridhar for the info. 3to2's version info has been PEP386-ified, and the latest version can always be found at http://pypi.python.org/pypi/3to2/ (the previous link will now generate an error). --Joe On Mon, Aug 31, 2009 at 3:25 PM, Brett Cannon wrote: > On Mon, Aug 31, 2009 at 12:18, Sridhar > Ratnakumar wrote: > > On Wed, 26 Aug 2009 15:55:54 -0700, Joe Amenta wrote: > > > >> -- 3to2 is now registered with PyPI. Did I do it right? > > > >> http://pypi.python.org/pypi/3to2/0.1%20alpha%201 > > > > Please fix the version number to not contain any whitespace characters. > Also > > set the `version` argument in setup(..) in your setup.py. And then you > may > > want to use the `upload` command to upload the new tarball to PyPI. See > > http://wiki.python.org/moin/Distutils/Tutorial for more details. > > See PEP 386 (http://www.python.org/dev/peps/pep-0386/) for what the > current thinking on version numbers is. > > -Brett > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
