Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-16 Thread Guido van Rossum
Thanks for all the changes! On Fri, Apr 16, 2010 at 4:00 PM, Barry Warsaw wrote: > On Apr 15, 2010, at 08:01 PM, Guido van Rossum wrote: >>Hm. I wish there was a way to find out whether the bytecode (or >>whatever) actually *was* read from this file. __file__ in Python 2 >>supports this (though n

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-16 Thread Barry Warsaw
On Apr 16, 2010, at 11:52 AM, Guido van Rossum wrote: >To the contrary, it was invented by Barry and ought to be added to the >English language as a neologism. Actually, it's an Emacs invention! -Barry signature.asc Description: PGP signature ___ Pyt

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-16 Thread Barry Warsaw
On Apr 15, 2010, at 08:01 PM, Guido van Rossum wrote: >> Byte code files contain two 32-bit numbers followed by the marshaled > >big-endian Done. >> [2]_ code object.  The 32-bit numbers represent a magic number and a >> timestamp.  The magic number changes whenever Python changes the byte >> co

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-16 Thread Benjamin Peterson
2010/4/16 Barry Warsaw : > On Apr 15, 2010, at 08:01 PM, Guido van Rossum wrote: >>> This feature is targeted for Python 3.2, solving the problem for those >>> and all future versions.  It may be back-ported to Python 2.7. >> >>Is there time given that 2.7b1 was released? > > I think this would be

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-16 Thread Guido van Rossum
On Fri, Apr 16, 2010 at 11:09 AM, Terry Reedy wrote: > On 4/15/2010 11:01 PM, Guido van Rossum wrote: >>> >>> pyc files inside of the `__pycache__` directories contain a magic >>> identifier in their file names.  These are mnemonic tags for the >>> actual magic numbers used by the importer.  For e

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-16 Thread Terry Reedy
On 4/15/2010 11:01 PM, Guido van Rossum wrote: pyc files inside of the `__pycache__` directories contain a magic identifier in their file names. These are mnemonic tags for the actual magic numbers used by the importer. For example, in Python 3.2, we could use the hexlified [10]_ magic number a

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-16 Thread Barry Warsaw
On Apr 15, 2010, at 08:01 PM, Guido van Rossum wrote: >Comments inline. Nothing showstopping, mostly just spewing obscure >background information... > >Overall, congratulations! I'm fine with the implementation going in >and the PEP being marked as accepted as long as you get to the >clarification

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-15 Thread Steve Holden
Guido van Rossum wrote: [...] >> Implementation strategy >> === >> >> This feature is targeted for Python 3.2, solving the problem for those >> and all future versions. It may be back-ported to Python > > Is there time given that 2.7b1 was released? > I would hope we have le

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-15 Thread Guido van Rossum
Comments inline. Nothing showstopping, mostly just spewing obscure background information... Overall, congratulations! I'm fine with the implementation going in and the PEP being marked as accepted as long as you get to the clarifications I suggest below soon after. --Guido On Tue, Apr 13, 2010

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-14 Thread Barry Warsaw
On Apr 15, 2010, at 08:33 AM, Nick Coghlan wrote: >Brett Cannon wrote: >> And just a quick suggestion: can we standardize what >> imp.source_to_path() and friend are supposed to return if the >> interpreter doesn't support bytecode? I will probably have to rely on >> that for something so it would

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-14 Thread Nick Coghlan
Brett Cannon wrote: > And just a quick suggestion: can we standardize what > imp.source_to_path() and friend are supposed to return if the > interpreter doesn't support bytecode? I will probably have to rely on > that for something so it would be best to say now whether it should be > None or raise

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-14 Thread Barry Warsaw
On Apr 14, 2010, at 09:52 AM, Isaac Morland wrote: >I have one wording suggestion which I hope isn't bikeshedding: up above, I >think the sentence containing "pyc files will contain a magic string" >would be clearer if it made it clear that the file *names*, not (just?) >the file contents, will

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-14 Thread Nick Coghlan
Isaac Morland wrote: > I have one wording suggestion which I hope isn't bikeshedding: up above, > I think the sentence containing "pyc files will contain a magic string" > would be clearer if it made it clear that the file *names*, not (just?) > the file contents, will contain the magic tag. That'

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-14 Thread Isaac Morland
On Tue, 13 Apr 2010, Barry Warsaw wrote: I am attaching the latest revision of PEP 3147 to this message, which is also available here: http://www.python.org/dev/peps/pep-3147/ [] PEP: 3147 Title: PYC Repository Directories [] Further, pyc files will contain a magic string that diff

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-14 Thread Barry Warsaw
On Apr 13, 2010, at 04:44 PM, Guido van Rossum wrote: >Give me a couple of days; but I don't expect any problems given how >the earlier discussion went. If you didn't hear from me by Friday go >ahead and merge. Thanks Guido. -Barry signature.asc Description: PGP signature __

Re: [Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-13 Thread Guido van Rossum
On Tue, Apr 13, 2010 at 1:21 PM, Barry Warsaw wrote: > I am attaching the latest revision of PEP 3147 to this message, which is also > available here: > > http://www.python.org/dev/peps/pep-3147/ > > I think the PEP is ready for pronouncement, and the patch is pretty much ready > for merging into

[Python-Dev] PEP 3147 ready for pronouncement and merging

2010-04-13 Thread Barry Warsaw
I am attaching the latest revision of PEP 3147 to this message, which is also available here: http://www.python.org/dev/peps/pep-3147/ I think the PEP is ready for pronouncement, and the patch is pretty much ready for merging into py3k. The only thing that I can think of that is not implemented

Re: [Python-Dev] PEP 3147

2010-04-11 Thread Barry Warsaw
On Apr 11, 2010, at 02:39 AM, Nick Coghlan wrote: >Barry Warsaw wrote: >> I don't have strong feelings about this. I thought it would be handy for >> future Python's to have access to this, but then, without access to previous >> version magic numbers, it probably doesn't help much. And as you s

Re: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()

2010-04-11 Thread Georg Brandl
Am 11.04.2010 00:25, schrieb Barry Warsaw: > On Apr 10, 2010, at 08:28 PM, Georg Brandl wrote: > >>Am 10.04.2010 18:12, schrieb Guido van Rossum: >>> On Sat, Apr 10, 2010 at 8:58 AM, Barry Warsaw wrote: On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: >On Fri, Apr 9, 2010 at 3:

Re: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()

2010-04-10 Thread Barry Warsaw
On Apr 10, 2010, at 08:28 PM, Georg Brandl wrote: >Am 10.04.2010 18:12, schrieb Guido van Rossum: >> On Sat, Apr 10, 2010 at 8:58 AM, Barry Warsaw wrote: >>> On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: >>> On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: > Would it be better t

Re: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()

2010-04-10 Thread Georg Brandl
Am 10.04.2010 18:12, schrieb Guido van Rossum: > On Sat, Apr 10, 2010 at 8:58 AM, Barry Warsaw wrote: >> On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: >> >>>On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: Would it be better to name this one _PyImport_ExecCodeModuleExEx (with a

Re: [Python-Dev] PEP 3147

2010-04-10 Thread Nick Coghlan
Barry Warsaw wrote: > I don't have strong feelings about this. I thought it would be handy for > future Python's to have access to this, but then, without access to previous > version magic numbers, it probably doesn't help much. And as you say, CPython > won't know about alternative implementati

Re: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()

2010-04-10 Thread Guido van Rossum
On Sat, Apr 10, 2010 at 8:58 AM, Barry Warsaw wrote: > On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: > >>On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: >>> Would it be better to name this one _PyImport_ExecCodeModuleExEx (with >>> an underscore) so that we *don't* need to create an ExE

Re: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()

2010-04-10 Thread Barry Warsaw
On Apr 09, 2010, at 05:41 PM, Guido van Rossum wrote: >On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: >> Would it be better to name this one _PyImport_ExecCodeModuleExEx (with >> an underscore) so that we *don't* need to create an ExExEx version in >> future? (Sorry, Barry :-)) > >I don't care

Re: [Python-Dev] PEP 3147

2010-04-10 Thread Barry Warsaw
On Apr 09, 2010, at 11:05 PM, Antoine Pitrou wrote: >« Instead, this PEP proposes to add a mapping between internal magic numbers >and a user-friendly tag. Newer versions of Python can add to this mapping so >that all later Pythons know the mapping between tags and magic numbers. » > >The questi

Re: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()

2010-04-09 Thread Guido van Rossum
On Fri, Apr 9, 2010 at 3:54 PM, Paul Moore wrote: > On 9 April 2010 23:00, Barry Warsaw wrote: >> On Apr 09, 2010, at 02:52 PM, Guido van Rossum wrote: >> >>>It may be undocumented but it doesn't start with _ and it exists to >>>preserve backwards compatibility. So I recommend adding >>>PyImport_

Re: [Python-Dev] PEP 3147

2010-04-09 Thread Antoine Pitrou
Hello, I'm sorry to chime in a bit late about an aspect of the PEP, but I only wondered about it while reviewing your code :-) « Instead, this PEP proposes to add a mapping between internal magic numbers and a user-friendly tag. Newer versions of Python can add to this mapping so that all late

Re: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()

2010-04-09 Thread Paul Moore
On 9 April 2010 23:00, Barry Warsaw wrote: > On Apr 09, 2010, at 02:52 PM, Guido van Rossum wrote: > >>It may be undocumented but it doesn't start with _ and it exists to >>preserve backwards compatibility. So I recommend adding >>PyImport_ExecCodeModuleExEx(). > > Cool, thanks.  Now I can't wait

Re: [Python-Dev] PEP 3147 working implementation

2010-04-09 Thread Barry Warsaw
Patch set 4 has been uploaded to Rietveld: http://codereview.appspot.com/842043/show This includes some fixes for Windows and support for the __cached__ attribute on modules. While I need to do another pass through the PEP to make sure I've gotten everything, this code is very nearly feature com

Re: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()

2010-04-09 Thread Barry Warsaw
On Apr 09, 2010, at 02:52 PM, Guido van Rossum wrote: >It may be undocumented but it doesn't start with _ and it exists to >preserve backwards compatibility. So I recommend adding >PyImport_ExecCodeModuleExEx(). Cool, thanks. Now I can't wait for PyImport_ExecCodeModuleExExEx() :) -Barry sign

Re: [Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()

2010-04-09 Thread Guido van Rossum
It may be undocumented but it doesn't start with _ and it exists to preserve backwards compatibility. So I recommend adding PyImport_ExecCodeModuleExEx(). On Fri, Apr 9, 2010 at 1:24 PM, Barry Warsaw wrote: > I've run into a minor snag implementing the __cached__ attribute on imported > modules.

[Python-Dev] PEP 3147, __cached__, and PyImport_ExecCodeModuleEx()

2010-04-09 Thread Barry Warsaw
I've run into a minor snag implementing the __cached__ attribute on imported modules. From PEP 3147: As part of this PEP, we will add an `__cached__` attribute to modules, which will always point to the actual `pyc` file that was read or written. When the environment variable `$PYTHO

Re: [Python-Dev] PEP 3147 working implementation

2010-04-08 Thread Barry Warsaw
On Apr 06, 2010, at 09:57 PM, Benjamin Peterson wrote: >I've now added a review, too. As did Brett. Thanks! I've responded and will upload a new patch set as soon as I've verified the test suite passes. -Barry signature.asc Description: PGP signature _

Re: [Python-Dev] PEP 3147 working implementation

2010-04-06 Thread Benjamin Peterson
2010/4/6 Barry Warsaw : > On Apr 01, 2010, at 04:12 PM, Barry Warsaw wrote: > >>I now have a working implementation of PEP 3147 which passes all the existing, >>and new, tests.  I'm sure there's still work to do, but I think the branch >>is in good enough shape to start getting some feedback from p

Re: [Python-Dev] PEP 3147 working implementation

2010-04-06 Thread Barry Warsaw
On Apr 01, 2010, at 04:12 PM, Barry Warsaw wrote: >I now have a working implementation of PEP 3147 which passes all the existing, >and new, tests. I'm sure there's still work to do, but I think the branch >is in good enough shape to start getting some feedback from python-dev. Thanks to Antoine

Re: [Python-Dev] PEP 3147 working implementation

2010-04-01 Thread Barry Warsaw
On Apr 01, 2010, at 09:27 PM, Antoine Pitrou wrote: >If you want a review, perhaps you should post it to Rietveld. Good idea. http://codereview.appspot.com/842043/show I've never used Rietveld before so let me know if I need to do anything to invite you or otherwise make the review possible. I

Re: [Python-Dev] PEP 3147 working implementation

2010-04-01 Thread Antoine Pitrou
Barry Warsaw python.org> writes: > > You can also view a live diff online: > > https://code.launchpad.net/~barry/python/pep3147/+merge/22648 If you want a review, perhaps you should post it to Rietveld. Regards Antoine. ___ Python-Dev mailing list

[Python-Dev] PEP 3147 working implementation

2010-04-01 Thread Barry Warsaw
I now have a working implementation of PEP 3147 which passes all the existing, and new, tests. I'm sure there's still work to do, but I think the branch is in good enough shape to start getting some feedback from python-dev. You can grab the changes in several ways. If you have Bazaar, you can c

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-24 Thread Greg Ewing
Isaac Morland wrote: the benefit to me and to Greg and to others writing .py code is that our directories will contain *.py and __pycache__, rather than *.py and *.pyc. So it will be much easier to see what is actually there. Yes. When using MacOSX I do most of my work using the Finder's colu

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-24 Thread Nick Coghlan
Russell E. Owen wrote: > This in turn implies that we may have to give up some support for > dragging python modules into site-packages, e.g. not generate .pyc files > for such modules. At least if we go that route it will mostly affect > power users, who can presumably cope. And when someone d

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Antoine Pitrou
Le mardi 23 mars 2010 à 20:50 -0400, Isaac Morland a écrit : > > I'm sure Greg will jump in if I'm wrong about what he is saying, but the > benefit to me and to Greg and to others writing .py code is that our > directories will contain *.py and __pycache__, rather than *.py and *.pyc. > So it w

Re: [Python-Dev] PEP 3147, __pycache__ directorie s and umask

2010-03-23 Thread Isaac Morland
On Tue, 23 Mar 2010, Antoine Pitrou wrote: Greg Ewing canterbury.ac.nz> writes: The main point of the __pycache__ proposal is to solve the needs of Ubuntu/Debian packagers. If you are developing (rather than deploying or building packages), you shouldn't have these needs AFAICT. Maybe it's o

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Ben Finney
Antoine Pitrou writes: Steven D'Aprano writes: > On Wed, 24 Mar 2010 12:35:43 am Ben Finney wrote: > > > On 23.03.2010 02:28, Ben Finney wrote: > > > > http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html > > > >#VARCACHEAPPLICATIONCACHEDATA> > > > > > > > > This would suggest that Pyth

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Barry Warsaw
On Mar 23, 2010, at 12:49 PM, Russell E. Owen wrote: >If .pyc files are to be shared, it seems essential to (by default) >generate them at install time and make them read-only for unprivileged >users. I think in practice this is what's almost always going to happen for system Python source, eit

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Barry Warsaw
On Mar 24, 2010, at 12:35 AM, Ben Finney wrote: >So what? There's no implication that data-which-happens-to-be-code is >unsuitable for storage in ‘/var/cache/foo/’. Easily-regenerated Python >byte code for caching meets the description quite well, AFAICT. pyc files don't go there now, so why woul

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Barry Warsaw
On Mar 22, 2010, at 08:33 PM, Antoine Pitrou wrote: >Well, precisely. That's why I suggest that creating the __pycache__ >directories be done *at install time* (or packaging time), and not via the >core import machinery (that is, not at import time). That is, when you *know* >you are the right use

Re: [Python-Dev] PEP 3147, __pycache__ directorie s and umask

2010-03-23 Thread Antoine Pitrou
Greg Ewing canterbury.ac.nz> writes: > > The main point of the __pycache__ proposal is to solve the needs of > > Ubuntu/Debian packagers. If you are developing (rather than deploying or > > building packages), you shouldn't have these needs AFAICT. > > Maybe it's one point, but I'm not sure it's

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Greg Ewing
Russell E. Owen wrote: If .pyc files are to be shared, it seems essential to (by default) generate them at install time and make them read-only for unprivileged users. This in turn implies that we may have to give up some support for dragging python modules into site-packages No, I don't t

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Greg Ewing
Antoine Pitrou wrote: The main point of the __pycache__ proposal is to solve the needs of Ubuntu/Debian packagers. If you are developing (rather than deploying or building packages), you shouldn't have these needs AFAICT. Maybe it's one point, but I'm not sure it's the *main* one. Personally I

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Russell E. Owen
In article <4ba80418.6030...@canterbury.ac.nz>, Greg Ewing wrote: > Antoine Pitrou wrote: > > > In light of this issue, I'm -0.5 on __pycache__ becoming the default > > caching > > mechanism. The directory ownership/permissions issue is too much of a mess, > > especially for Web applications (

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Steven D'Aprano
On Wed, 24 Mar 2010 12:35:43 am Ben Finney wrote: > Matthias Klose writes: > > On 23.03.2010 02:28, Ben Finney wrote: > > > Perhaps also of note is that the FHS recommends systems use > > > ‘/var/cache/foo/’ for cached data from applications: > > > > > > /var/cache : Application cache data >

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Nick Coghlan
Antoine Pitrou wrote: >> or if a user installs by dragging into >> site-packages instead of using an installer? > > Well... do people actually do this? Yes. We do it all the time with unpackaged only-for-internal-use Python code. I wouldn't expect it to work with random packages downloaded from

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-23 Thread Ben Finney
Matthias Klose writes: > On 23.03.2010 02:28, Ben Finney wrote: > > Perhaps also of note is that the FHS recommends systems use > > ‘/var/cache/foo/’ for cached data from applications: > > > > /var/cache : Application cache data > > > > Purpose > > > > /var/cache is intended for ca

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-22 Thread Matthias Klose
On 23.03.2010 02:28, Ben Finney wrote: Antoine Pitrou writes: Barry Warsaw python.org> writes: When Python is being installed, either by a from-source 'make install' or by the distro packager, then you'd expect the umask not to be insane. In the latter case, it's a bug and in the former ca

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-22 Thread Ben Finney
Antoine Pitrou writes: > Barry Warsaw python.org> writes: > > > > When Python is being installed, either by a from-source 'make > > install' or by the distro packager, then you'd expect the umask not > > to be insane. In the latter case, it's a bug and in the former case > > you screwed up so y

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-22 Thread Cameron Simpson
On 23Mar2010 11:40, I wrote: | | * Raise an exception or refuse to create __pycache__ if it's not writable | | (again, by whom?) | | -3 | Bleah. My python program won't run because an obscure (to the user) | directory had unusual permissions? Clarification: I'm -3 on the exception. Silent failu

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-22 Thread Cameron Simpson
On 22Mar2010 09:56, Barry Warsaw wrote: | I have a pretty good start on PEP 3147 implementation [1], but I've | encountered a situation that I'd like to get some feedback on. Here's the | test case illustrating the problem. From test_import.py: | | def test_writable_directory(self): |

Re: [Python-Dev] PEP 3147, __pycache__ directorie s and umask

2010-03-22 Thread Antoine Pitrou
Greg Ewing canterbury.ac.nz> writes: > > Doesn't the existing .pyc mechanism have the same problem? Seems > to me it's just as insecure to allow the Apache user to create > .pyc files, since an attacker could overwrite them with arbitrary > bytecode. The problem is that you can't delete the __py

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-22 Thread Greg Ewing
Antoine Pitrou wrote: In light of this issue, I'm -0.5 on __pycache__ becoming the default caching mechanism. The directory ownership/permissions issue is too much of a mess, especially for Web applications (think __pycache__ files created by the Apache user). Doesn't the existing .pyc mechani

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-22 Thread Brett Cannon
On Mon, Mar 22, 2010 at 06:56, Barry Warsaw wrote: > I have a pretty good start on PEP 3147 implementation [1], but I've > encountered a situation that I'd like to get some feedback on.  Here's the > test case illustrating the problem.  From test_import.py: > >    def test_writable_directory(self)

Re: [Python-Dev] PEP 3147, __pycache__ directorie s and umask

2010-03-22 Thread Antoine Pitrou
Barry Warsaw python.org> writes: > > When Python is being installed, either by a from-source 'make install' or by > the distro packager, then you'd expect the umask not to be insane. In the > latter case, it's a bug and in the former case you screwed up so you should be > able to delete and rein

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-22 Thread Barry Warsaw
On Mar 22, 2010, at 02:30 PM, Antoine Pitrou wrote: >Barry Warsaw python.org> writes: >> >> * Tough luck >> * Force the umask so that the directory is writable, but then the question >> is, >> by whom? ugo+w or something less? >> * Copy the permissions from the parent directory and ignore um

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-22 Thread Martin v. Löwis
> refuse to create __pycache__ if I can't create it with the same ownership > and permissions as the parent directory (and raise an ImportWarning). I don't think an ImportWarning should be raised: AFAICT, we are not raising one, either, when the pyc file cannot be created (and it may well be t

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-22 Thread Martin v. Löwis
> Or maybe you have a better idea? What's the equivalent situation on Windows > and how would things work there? On Windows, this problem is easy: create the directory with no specification of an ACL, and it will (usually) inherit the ACL of the parent directory. IOW, the same users will have wri

Re: [Python-Dev] PEP 3147, __pycache__ directorie s and umask

2010-03-22 Thread Antoine Pitrou
Barry Warsaw python.org> writes: > > I'm not sure what the right answer is. Some possible choices: > > * Tough luck > * Force the umask so that the directory is writable, but then the question is, > by whom? ugo+w or something less? > * Copy the permissions from the parent directory and igno

[Python-Dev] PEP 3147, __pycache__ directories and umask

2010-03-22 Thread Barry Warsaw
I have a pretty good start on PEP 3147 implementation [1], but I've encountered a situation that I'd like to get some feedback on. Here's the test case illustrating the problem. From test_import.py: def test_writable_directory(self): # The umask is not conducive to creating a writabl

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-09 Thread Nick Coghlan
Terry Reedy wrote: > Definitely. I have even wondered whether it would be possible to cache > not just the bytecode for initializing a module, but also the > initialized module itself (perhaps minus the name bindings for other > imported modules). Not easily, since running the module may have othe

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-08 Thread Terry Reedy
On 2/8/2010 7:54 AM, Nick Coghlan wrote: Ron Adam wrote: To tell the truth in most cases I hardly notice the extra time the first run takes compared to later runs with the precompiled byte code. Yes it may be a few seconds at start up, but after that it's usually not a big part of the execution

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-08 Thread Nick Coghlan
Ron Adam wrote: > To tell the truth in most cases I hardly notice the extra time the first > run takes compared to later runs with the precompiled byte code. Yes it > may be a few seconds at start up, but after that it's usually not a big > part of the execution time. Hmmm, I wonder if there's a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Ron Adam
Barry Warsaw wrote: On Jan 31, 2010, at 01:06 PM, Ron Adam wrote: With a single cache directory, we could have an option to force writing bytecode to a desired location. That might be useful on it's own for creating runtime bytecode only installations for installers. One important reason

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Guido van Rossum
On Sun, Feb 7, 2010 at 12:23 PM, Brett Cannon wrote: > On Sun, Feb 7, 2010 at 10:44, Barry Warsaw wrote: >> On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: >> >>>The conflict is purely that PEP 3147 proposes the new behavior to be >>>optional, and adds a flag (-R) and an environment variabl

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Brett Cannon
On Sun, Feb 7, 2010 at 10:44, Barry Warsaw wrote: > On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: > >>The conflict is purely that PEP 3147 proposes the new behavior to be >>optional, and adds a flag (-R) and an environment variable >>($PYTHONPYR) to change it. I presume Barry is proposing

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Guido van Rossum
On Sun, Feb 7, 2010 at 10:17 AM, Barry Warsaw wrote: > On Jan 31, 2010, at 11:34 PM, Nick Coghlan wrote: > >>I must admit I quite like the __pyr__ directory approach as well. Since >>the interpreter knows the suffix it is looking for, names shouldn't >>conflict. Using a single directory allows the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread M.-A. Lemburg
Barry Warsaw wrote: > On Feb 03, 2010, at 11:59 AM, M.-A. Lemburg wrote: > >> How about using an optionally relative cache dir setting to let >> the user decide ? > > Why do we need that level of flexibility? It's very easy to implement (see the code I posted) and gives you a lot of control with

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 07, 2010, at 05:59 PM, Michael Foord wrote: >On 07/02/2010 17:48, Barry Warsaw wrote: >> [snip...] >>> And I propose not to disturb this in 2.7, at least not by default. I'm >>> fine though with a flag or distro-overridable config setting to change >>> this behavior. >>> >> Cool. I'm

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: >The conflict is purely that PEP 3147 proposes the new behavior to be >optional, and adds a flag (-R) and an environment variable >($PYTHONPYR) to change it. I presume Barry is proposing this out of >fear that the new behavior might upset someb

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 06, 2010, at 04:02 PM, Guido van Rossum wrote: >On Sat, Feb 6, 2010 at 3:28 PM, Barry Warsaw wrote: >> On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: >> >>>It's an interesting challenge to write the file in such a way that >>>it's safe for a reader and writer to co-exist. Like Brett, I

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 01, 2010, at 08:26 AM, Tim Delaney wrote: >The pyc/pyo files are just an optimisation detail, and are essentially >temporary. Given that, if they were to live in a single directory, to me it >seems obvious that the default location for that should be in the system >temporary directory. I an

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Jan 31, 2010, at 11:34 PM, Nick Coghlan wrote: >I must admit I quite like the __pyr__ directory approach as well. Since >the interpreter knows the suffix it is looking for, names shouldn't >conflict. Using a single directory allows the name to be less cryptic, >too (e.g. __pycache__). Somethin

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Jan 31, 2010, at 08:10 PM, Silke von Bargen wrote: >Martin v. Löwis schrieb: >> There is also the issue of race conditions with multiple simultaneous >> accesses. The original format for the PEP had race conditions for >> multiple simultaneous writers; ZIP will also have race conditions for >>

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Michael Foord
On 07/02/2010 17:48, Barry Warsaw wrote: [snip...] And I propose not to disturb this in 2.7, at least not by default. I'm fine though with a flag or distro-overridable config setting to change this behavior. Cool. I'm not sure this is absolutely necessary for Debian/Ubuntu, so I'll call

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Jan 31, 2010, at 01:06 PM, Ron Adam wrote: >With a single cache directory, we could have an option to force writing >bytecode to a desired location. That might be useful on it's own for >creating runtime bytecode only installations for installers. One important reason for wanting to keep th

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 04, 2010, at 03:00 PM, Glenn Linderman wrote: >When a PEP 3147 (if modified by my suggestion) version of Python runs, >and the directory doesn't exist, and it wants to create a .pyc, it would >create the directory, and put the .pyc there. Sort of just like how it >creates .pyc files, no

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 06, 2010, at 02:20 PM, Guido van Rossum wrote: >> Upon further reflection, I agree.  __file__ also points to the source in >> Python 2.7. > >Not in the 2.7 svn repo I have access to. It still points to the .pyc >file if it was used. Ah, I was fooled by a missing pyc file. Run it a second

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Nick Coghlan
exar...@twistedmatrix.com wrote: > On 08:21 pm, ba...@python.org wrote: >> On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: >>> Can you clarify? In Python 3, __file__ always points to the source. >>> Clearly that is the way of the future. For 99.99% of uses of __file__, >>> if it suddenly neve

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Ben Finney
Nick Coghlan writes: > The more decoupled they are, the harder it is to manually find the > bytecode file. Okay. So it's not so much about “predictable”, but rather about “predictable by a human without too much cognitive effort”. I can see value in that, though it's best to be explicit that th

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 5:10 PM, Nick Coghlan wrote: > Ben Finney wrote: >> Right; I don't see who would disagree with that. I don't see any >> conflict between “decouple compiled bytecode file locations from source >> file locations” versus “predictable location for the compiled bytecode >> files”

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Nick Coghlan
Ben Finney wrote: > Right; I don't see who would disagree with that. I don't see any > conflict between “decouple compiled bytecode file locations from source > file locations” versus “predictable location for the compiled bytecode > files”. The more decoupled they are, the harder it is to manuall

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 4:27 PM, Ben Finney wrote: > Barry Warsaw writes: >> I agree. I'd prefer to have a predictable place for the cached files, >> independent of having to run Python to tell you where that is. > > Right; I don't see who would disagree with that. I don't see any > conflict betwe

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Ben Finney
Barry Warsaw writes: > On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote: > > >It's also the case that having to run Python to manage my own > >filesystem would very annoying. […] Files that are problematic wouldn't need Python to manage any more than currently. The suggestion was just that, a su

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 3:28 PM, Barry Warsaw wrote: > On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: > >>It's an interesting challenge to write the file in such a way that >>it's safe for a reader and writer to co-exist. Like Brett, I >>considered an append-only scheme, but one needs to handle

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 01, 2010, at 11:28 PM, Martin v. Löwis wrote: >So what would you do for concurrent writers, then? The current >implementation relies on creat(O_EXCL) to be atomic, so a second >writer would just fail. This is but the only IO operation that is >guaranteed to be atomic (along with mkdir(2)),

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: >It's an interesting challenge to write the file in such a way that >it's safe for a reader and writer to co-exist. Like Brett, I >considered an append-only scheme, but one needs to handle the case >where the bytecode for a particular magic number

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Jan 31, 2010, at 11:04 AM, Raymond Hettinger wrote: >> It does this by >> allowing many different byte compilation files (.pyc files) to be >> co-located with the Python source file (.py file). > >It would be nice if all the compilation files could be tucked >into one single zipfile per dire

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 12:21 PM, Barry Warsaw wrote: > On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: >>Can you clarify? In Python 3, __file__ always points to the source. >>Clearly that is the way of the future. For 99.99% of uses of __file__, >>if it suddenly never pointed to a .pyc file

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread exarkun
On 08:21 pm, ba...@python.org wrote: On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: Can you clarify? In Python 3, __file__ always points to the source. Clearly that is the way of the future. For 99.99% of uses of __file__, if it suddenly never pointed to a .pyc file any more (even if one

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 09:26 AM, Floris Bruynooghe wrote: >On Wed, Feb 03, 2010 at 06:14:44PM +1100, Ben Finney wrote: >> I don't understand the distinction you're making between those two >> options. Can you explain what you mean by each of “siblings” and >> “folder-per-folder”? > >sibilings: the or

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote: >It's also the case that having to run Python to manage my own filesystem >would very annoying. If a dev has a broken .pyc that prevents the >affected Python build from even starting how are they meant to use the >nonfunctioning interpreter to find

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 11:59 AM, M.-A. Lemburg wrote: >How about using an optionally relative cache dir setting to let >the user decide ? Why do we need that level of flexibility? -Barry signature.asc Description: PGP signature ___ Python-Dev mailing li

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 05, 2010, at 07:37 PM, Nick Coghlan wrote: >Brett Cannon wrote: >> Does code exist out there where people are constructing bytecode from >> multiple files for a single module? > >I'm quite prepared to call YAGNI on that idea and just return a 2-tuple >of source filename and compiled filena

  1   2   3   >