Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-09 Thread Nick Coghlan
Terry Reedy wrote: > Definitely. I have even wondered whether it would be possible to cache > not just the bytecode for initializing a module, but also the > initialized module itself (perhaps minus the name bindings for other > imported modules). Not easily, since running the module may have othe

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-08 Thread Terry Reedy
On 2/8/2010 7:54 AM, Nick Coghlan wrote: Ron Adam wrote: To tell the truth in most cases I hardly notice the extra time the first run takes compared to later runs with the precompiled byte code. Yes it may be a few seconds at start up, but after that it's usually not a big part of the execution

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-08 Thread Nick Coghlan
Ron Adam wrote: > To tell the truth in most cases I hardly notice the extra time the first > run takes compared to later runs with the precompiled byte code. Yes it > may be a few seconds at start up, but after that it's usually not a big > part of the execution time. Hmmm, I wonder if there's a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Ron Adam
Barry Warsaw wrote: On Jan 31, 2010, at 01:06 PM, Ron Adam wrote: With a single cache directory, we could have an option to force writing bytecode to a desired location. That might be useful on it's own for creating runtime bytecode only installations for installers. One important reason

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Guido van Rossum
On Sun, Feb 7, 2010 at 12:23 PM, Brett Cannon wrote: > On Sun, Feb 7, 2010 at 10:44, Barry Warsaw wrote: >> On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: >> >>>The conflict is purely that PEP 3147 proposes the new behavior to be >>>optional, and adds a flag (-R) and an environment variabl

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Brett Cannon
On Sun, Feb 7, 2010 at 10:44, Barry Warsaw wrote: > On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: > >>The conflict is purely that PEP 3147 proposes the new behavior to be >>optional, and adds a flag (-R) and an environment variable >>($PYTHONPYR) to change it. I presume Barry is proposing

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Guido van Rossum
On Sun, Feb 7, 2010 at 10:17 AM, Barry Warsaw wrote: > On Jan 31, 2010, at 11:34 PM, Nick Coghlan wrote: > >>I must admit I quite like the __pyr__ directory approach as well. Since >>the interpreter knows the suffix it is looking for, names shouldn't >>conflict. Using a single directory allows the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread M.-A. Lemburg
Barry Warsaw wrote: > On Feb 03, 2010, at 11:59 AM, M.-A. Lemburg wrote: > >> How about using an optionally relative cache dir setting to let >> the user decide ? > > Why do we need that level of flexibility? It's very easy to implement (see the code I posted) and gives you a lot of control with

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 07, 2010, at 05:59 PM, Michael Foord wrote: >On 07/02/2010 17:48, Barry Warsaw wrote: >> [snip...] >>> And I propose not to disturb this in 2.7, at least not by default. I'm >>> fine though with a flag or distro-overridable config setting to change >>> this behavior. >>> >> Cool. I'm

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote: >The conflict is purely that PEP 3147 proposes the new behavior to be >optional, and adds a flag (-R) and an environment variable >($PYTHONPYR) to change it. I presume Barry is proposing this out of >fear that the new behavior might upset someb

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 06, 2010, at 04:02 PM, Guido van Rossum wrote: >On Sat, Feb 6, 2010 at 3:28 PM, Barry Warsaw wrote: >> On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: >> >>>It's an interesting challenge to write the file in such a way that >>>it's safe for a reader and writer to co-exist. Like Brett, I

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 01, 2010, at 08:26 AM, Tim Delaney wrote: >The pyc/pyo files are just an optimisation detail, and are essentially >temporary. Given that, if they were to live in a single directory, to me it >seems obvious that the default location for that should be in the system >temporary directory. I an

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Jan 31, 2010, at 11:34 PM, Nick Coghlan wrote: >I must admit I quite like the __pyr__ directory approach as well. Since >the interpreter knows the suffix it is looking for, names shouldn't >conflict. Using a single directory allows the name to be less cryptic, >too (e.g. __pycache__). Somethin

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Jan 31, 2010, at 08:10 PM, Silke von Bargen wrote: >Martin v. Löwis schrieb: >> There is also the issue of race conditions with multiple simultaneous >> accesses. The original format for the PEP had race conditions for >> multiple simultaneous writers; ZIP will also have race conditions for >>

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Michael Foord
On 07/02/2010 17:48, Barry Warsaw wrote: [snip...] And I propose not to disturb this in 2.7, at least not by default. I'm fine though with a flag or distro-overridable config setting to change this behavior. Cool. I'm not sure this is absolutely necessary for Debian/Ubuntu, so I'll call

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Jan 31, 2010, at 01:06 PM, Ron Adam wrote: >With a single cache directory, we could have an option to force writing >bytecode to a desired location. That might be useful on it's own for >creating runtime bytecode only installations for installers. One important reason for wanting to keep th

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 04, 2010, at 03:00 PM, Glenn Linderman wrote: >When a PEP 3147 (if modified by my suggestion) version of Python runs, >and the directory doesn't exist, and it wants to create a .pyc, it would >create the directory, and put the .pyc there. Sort of just like how it >creates .pyc files, no

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-07 Thread Barry Warsaw
On Feb 06, 2010, at 02:20 PM, Guido van Rossum wrote: >> Upon further reflection, I agree.  __file__ also points to the source in >> Python 2.7. > >Not in the 2.7 svn repo I have access to. It still points to the .pyc >file if it was used. Ah, I was fooled by a missing pyc file. Run it a second

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Nick Coghlan
exar...@twistedmatrix.com wrote: > On 08:21 pm, ba...@python.org wrote: >> On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: >>> Can you clarify? In Python 3, __file__ always points to the source. >>> Clearly that is the way of the future. For 99.99% of uses of __file__, >>> if it suddenly neve

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Ben Finney
Nick Coghlan writes: > The more decoupled they are, the harder it is to manually find the > bytecode file. Okay. So it's not so much about “predictable”, but rather about “predictable by a human without too much cognitive effort”. I can see value in that, though it's best to be explicit that th

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 5:10 PM, Nick Coghlan wrote: > Ben Finney wrote: >> Right; I don't see who would disagree with that. I don't see any >> conflict between “decouple compiled bytecode file locations from source >> file locations” versus “predictable location for the compiled bytecode >> files”

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Nick Coghlan
Ben Finney wrote: > Right; I don't see who would disagree with that. I don't see any > conflict between “decouple compiled bytecode file locations from source > file locations” versus “predictable location for the compiled bytecode > files”. The more decoupled they are, the harder it is to manuall

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 4:27 PM, Ben Finney wrote: > Barry Warsaw writes: >> I agree. I'd prefer to have a predictable place for the cached files, >> independent of having to run Python to tell you where that is. > > Right; I don't see who would disagree with that. I don't see any > conflict betwe

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Ben Finney
Barry Warsaw writes: > On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote: > > >It's also the case that having to run Python to manage my own > >filesystem would very annoying. […] Files that are problematic wouldn't need Python to manage any more than currently. The suggestion was just that, a su

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 3:28 PM, Barry Warsaw wrote: > On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: > >>It's an interesting challenge to write the file in such a way that >>it's safe for a reader and writer to co-exist. Like Brett, I >>considered an append-only scheme, but one needs to handle

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 01, 2010, at 11:28 PM, Martin v. Löwis wrote: >So what would you do for concurrent writers, then? The current >implementation relies on creat(O_EXCL) to be atomic, so a second >writer would just fail. This is but the only IO operation that is >guaranteed to be atomic (along with mkdir(2)),

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: >It's an interesting challenge to write the file in such a way that >it's safe for a reader and writer to co-exist. Like Brett, I >considered an append-only scheme, but one needs to handle the case >where the bytecode for a particular magic number

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Jan 31, 2010, at 11:04 AM, Raymond Hettinger wrote: >> It does this by >> allowing many different byte compilation files (.pyc files) to be >> co-located with the Python source file (.py file). > >It would be nice if all the compilation files could be tucked >into one single zipfile per dire

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Guido van Rossum
On Sat, Feb 6, 2010 at 12:21 PM, Barry Warsaw wrote: > On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: >>Can you clarify? In Python 3, __file__ always points to the source. >>Clearly that is the way of the future. For 99.99% of uses of __file__, >>if it suddenly never pointed to a .pyc file

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread exarkun
On 08:21 pm, ba...@python.org wrote: On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: Can you clarify? In Python 3, __file__ always points to the source. Clearly that is the way of the future. For 99.99% of uses of __file__, if it suddenly never pointed to a .pyc file any more (even if one

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 09:26 AM, Floris Bruynooghe wrote: >On Wed, Feb 03, 2010 at 06:14:44PM +1100, Ben Finney wrote: >> I don't understand the distinction you're making between those two >> options. Can you explain what you mean by each of “siblings” and >> “folder-per-folder”? > >sibilings: the or

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote: >It's also the case that having to run Python to manage my own filesystem >would very annoying. If a dev has a broken .pyc that prevents the >affected Python build from even starting how are they meant to use the >nonfunctioning interpreter to find

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 11:59 AM, M.-A. Lemburg wrote: >How about using an optionally relative cache dir setting to let >the user decide ? Why do we need that level of flexibility? -Barry signature.asc Description: PGP signature ___ Python-Dev mailing li

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 05, 2010, at 07:37 PM, Nick Coghlan wrote: >Brett Cannon wrote: >> Does code exist out there where people are constructing bytecode from >> multiple files for a single module? > >I'm quite prepared to call YAGNI on that idea and just return a 2-tuple >of source filename and compiled filena

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-06 Thread Barry Warsaw
On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: >Can you clarify? In Python 3, __file__ always points to the source. >Clearly that is the way of the future. For 99.99% of uses of __file__, >if it suddenly never pointed to a .pyc file any more (even if one >existed) that would be just fine. S

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-05 Thread Nick Coghlan
Brett Cannon wrote: > If we add a new method like get_filenames(), I would suggest going > with Antoine's suggestion of a tuple for __compiled__ (allowing > loaders to indicate that they actually constructed the runtime > bytecode from multiple cached files on-disk). > > > Does code exist out the

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Glenn Linderman
On approximately 2/4/2010 2:28 PM, came the following characters from the keyboard of Eric Smith: Glenn Linderman wrote: On approximately 1/30/2010 4:00 PM, came the following characters from the keyboard of Barry Warsaw: When the Python executable is given a `-R` flag, or the environment varia

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Eric Smith
Glenn Linderman wrote: On approximately 1/30/2010 4:00 PM, came the following characters from the keyboard of Barry Warsaw: When the Python executable is given a `-R` flag, or the environment variable `$PYTHONPYR` is set, then Python will create a `foo.pyr` directory and write a `pyc` file to th

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Brett Cannon
On Thu, Feb 4, 2010 at 13:51, Nick Coghlan wrote: > Brett Cannon wrote: > > My thinking is we deprecate get_filename() and introduce some new method > > that returns a two-item tuple (get_paths?). First item is where the > > source should be, and the second is where the bytecode is if it exists >

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Nick Coghlan
Glenn Linderman wrote: > Alt. C... source-file-dir/__pyr_version__, each Python version with > different bytecode would have some sort of version string or magic > number that identifies it, and would look only in that directory for its > .pyc/.pyo files. I prefer C for 4 reasons: 1) easier to blo

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Nick Coghlan
Brett Cannon wrote: > My thinking is we deprecate get_filename() and introduce some new method > that returns a two-item tuple (get_paths?). First item is where the > source should be, and the second is where the bytecode is if it exists > (else it's None). Putting both calculations into a single m

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Glenn Linderman
On approximately 1/30/2010 4:00 PM, came the following characters from the keyboard of Barry Warsaw: When the Python executable is given a `-R` flag, or the environment variable `$PYTHONPYR` is set, then Python will create a `foo.pyr` directory and write a `pyc` file to that directory with the he

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-04 Thread Brett Cannon
On Wed, Feb 3, 2010 at 13:33, "Martin v. Löwis" wrote: > Guido van Rossum wrote: > > On Wed, Feb 3, 2010 at 12:47 PM, Nick Coghlan > wrote: > >> On the issue of __file__, I'd suggesting not being too hasty in > >> deprecating that in favour of __source__. While I can see a lot of value > >> in h

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Ben Finney
Thanks for the explanation. Nick Coghlan writes: > Being able to get rid of the existing .pyc/.pyo clutter at the same > time is just a bonus. Okay. I maintain (unsurprisingly) that replacing it with subdirectory clutter is a poor bargain. But I have nothing new to add on that score for now. -

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Ben Finney wrote: > Nick Coghlan writes: > >> P.S. Translation of the double negative: I don't find any of the >> solutions, even the current .pyc/.pyo approach, to be particularly >> elegant, so I can't really say I like any of them in an absolute >> sense. However, having a single cache folder

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Guido van Rossum wrote: > On Wed, Feb 3, 2010 at 12:47 PM, Nick Coghlan wrote: >> On the issue of __file__, I'd suggesting not being too hasty in >> deprecating that in favour of __source__. While I can see a lot of value >> in having it point to the source file more often with a different >> attr

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Martin v. Löwis
Guido van Rossum wrote: > On Wed, Feb 3, 2010 at 12:47 PM, Nick Coghlan wrote: >> On the issue of __file__, I'd suggesting not being too hasty in >> deprecating that in favour of __source__. While I can see a lot of value >> in having it point to the source file more often with a different >> attr

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Guido van Rossum
On Wed, Feb 3, 2010 at 12:47 PM, Nick Coghlan wrote: > On the issue of __file__, I'd suggesting not being too hasty in > deprecating that in favour of __source__. While I can see a lot of value > in having it point to the source file more often with a different > attribute that points to the cache

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Ben Finney
Nick Coghlan writes: > P.S. Translation of the double negative: I don't find any of the > solutions, even the current .pyc/.pyo approach, to be particularly > elegant, so I can't really say I like any of them in an absolute > sense. However, having a single cache folder inside each Python source

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Brett Cannon
On Wed, Feb 3, 2010 at 05:27, Nick Coghlan wrote: > Glenn Linderman wrote: >> On approximately 2/2/2010 7:05 PM, came the following characters from >>  the keyboard of Guido van Rossum: >>> On Tue, Feb 2, 2010 at 5:41 PM, Glenn >>> Linderman  wrote: Agreed.  But in reading that, it somehow tr

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Barry Warsaw wrote: > On Feb 03, 2010, at 11:31 PM, Nick Coghlan wrote: > >> Having a lookup dictionary from Python version + C API magic numbers to >> the magic strings used in cache filenames in the import engine shouldn't >> be too tricky. I'll admit it wasn't until the thread had already been

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Barry Warsaw
On Feb 03, 2010, at 12:57 PM, Antoine Pitrou wrote: >How about doing measurements /with the current implementation/? Everyone >seems to worry about stat() calls but there doesn't seem to be any figures to >evaluate their significance. Yes, very good idea, if for no other reason than to give us a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Barry Warsaw
On Feb 03, 2010, at 11:31 PM, Nick Coghlan wrote: >Having a lookup dictionary from Python version + C API magic numbers to >the magic strings used in cache filenames in the import engine shouldn't >be too tricky. I'll admit it wasn't until the thread had already been >going for a while that I real

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Barry Warsaw wrote: > Encoding the magic number in the file name under .pyr would I thought make the > look up scheme more efficient since the import machinery can craft the file > name directly. I agree it's not very human friendly because nobody really > knows which magic numbers are associated

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Glenn Linderman wrote: > On approximately 2/2/2010 7:05 PM, came the following characters from > the keyboard of Guido van Rossum: >> On Tue, Feb 2, 2010 at 5:41 PM, Glenn >> Linderman wrote: >>> Agreed. But in reading that, it somehow triggered a question: >>> does zipimport only work for zipf

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Bob Ippolito wrote: > I like this option as well, but why not just name the directory .pyc > instead of __pyr__ or .pyr? That way people probably won't even have > to reconfigure their tools to ignore it :) This actually came up in another part of the thread. The conclusion was that, since the cac

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Floris Bruynooghe wrote: > Personally I'm +1 on the folder-per-folder option. Of all the proposed options, I also dislike the SVN/CVS style folder structure the least ;) Cheers, Nick. P.S. Translation of the double negative: I don't find any of the solutions, even the current .pyc/.pyo approach,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Nick Coghlan
Ben Finney wrote: > I don't think keeping the cache files in a mass of intertwingled extra > subdirectories is the way to solve that problem. That speaks, rather, to > the need for Python to be able to find the file on behalf of the user > and blow it away on request, so the user doesn't need to go

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Antoine Pitrou
Barry Warsaw python.org> writes: > > As to the question of sibling directories or folder-per-folder I think > performance issues should be the deciding factor. There are file system > limitations to consider (but also a wide variety of file systems in use). Do > the number of stat calls predomi

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread M.-A. Lemburg
> On 03/02/2010 06:50, Barry Warsaw wrote: >> As to the question of sibling directories or folder-per-folder I think >> performance issues should be the deciding factor. There are file system >> limitations to consider (but also a wide variety of file systems in >> use). Do >> the number of stat

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Michael Foord
On 03/02/2010 06:50, Barry Warsaw wrote: I have to say up front that I'm somewhat shocked at how quickly this thread has exploded! Since I'm sprinting this week, I haven't thoroughly read every message and won't have time tonight to answer every question, but I'll try to pick out some common ide

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Glenn Linderman
On approximately 2/2/2010 7:05 PM, came the following characters from the keyboard of Guido van Rossum: On Tue, Feb 2, 2010 at 5:41 PM, Glenn Linderman wrote: On approximately 2/2/2010 4:28 PM, came the following characters from the keyboard of Guido van Rossum: Argh. zipfiles are wa

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-03 Thread Floris Bruynooghe
On Wed, Feb 03, 2010 at 06:14:44PM +1100, Ben Finney wrote: > Barry Warsaw writes: > > > I suppose this is going to be very subjective, but in skimming the > > thread it seems like most people like putting the byte code cache > > artifacts in subdirectories (be they siblings or folder-per-folder)

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Ben Finney
Barry Warsaw writes: > If you have to manually blow away a particular pyc file, > folder-per-folder makes it much easier to find exactly what you want > to blow away without have to search up the file system, and then back > down again to find the pyc file to delete. How many ..'s does it take >

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Ben Finney
Barry Warsaw writes: > I suppose this is going to be very subjective, but in skimming the > thread it seems like most people like putting the byte code cache > artifacts in subdirectories (be they siblings or folder-per-folder). I don't understand the distinction you're making between those two

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 31, 2010, at 09:30 PM, Martin v. Löwis wrote: >If a single pyc folder is used, I think an additional __source__ >attribute would be needed to indicate what source file time stamp had >been checked (if any) to determine that the byte code file is current. This is a good point. __file__ is

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 31, 2010, at 12:36 PM, Georg Brandl wrote: >Not really -- much of the code I've seen that tries to guess the source >file name from a __file__ value just does something like this: > > if fname.lower().endswith(('.pyc', '.pyo')): fname = fname[:-1] > >That's not compatible with using .pyr,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 31, 2010, at 03:07 PM, Ben Finney wrote: >In other words, my understanding is that the current PEP would have the >following tree for an example project:: > >foo/ >__init__.py >__init__.pyr/ >deadbeef.pyc >decafbad.pyc >lorem.py >l

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 31, 2010, at 01:44 PM, Nick Coghlan wrote: >We deliberate don't document -U because its typical effect is "break the >world" - it makes all strings unicode in 2.x. As an aside, I think this should be documented *somewhere* other than just in import.c! I'd totally forgotten about it until

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
On Jan 30, 2010, at 11:21 PM, Vitor Bosshard wrote: >Why not: > >foo.py >foo.pyc # < 2.7 or < 3.2 >foo.27.pyc >foo.32.pyc >etc. Because this clutters the module's directory more than it does today, which I considered to be a negative factor. And as others have pointed out, there isn't a one-to-o

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Barry Warsaw
I have to say up front that I'm somewhat shocked at how quickly this thread has exploded! Since I'm sprinting this week, I haven't thoroughly read every message and won't have time tonight to answer every question, but I'll try to pick out some common ideas. I really appreciate everyone's input a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Bob Ippolito
On Sun, Jan 31, 2010 at 11:16 AM, Guido van Rossum wrote: > Whoa. This thread already exploded. I'm picking this message to > respond to because it reflects my own view after reading the PEP. > > On Sun, Jan 31, 2010 at 4:13 AM, Hanno Schlichting wrote: >> On Sun, Jan 31, 2010 at 1:03 PM, Simon C

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Guido van Rossum
On Tue, Feb 2, 2010 at 5:41 PM, Glenn Linderman wrote: > On approximately 2/2/2010 4:28 PM, came the following characters from the > keyboard of Guido van Rossum: >> >> Argh. zipfiles are way to complex to be writing. > > Agreed.  But in reading that, it somehow triggered a question: does > zipimp

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Glenn Linderman
On approximately 2/2/2010 4:28 PM, came the following characters from the keyboard of Guido van Rossum: Argh. zipfiles are way to complex to be writing. Agreed. But in reading that, it somehow triggered a question: does zipimport only work for zipfiles, or does it work for any archive format

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Guido van Rossum
Argh. zipfiles are way to complex to be writing. If you want to use zipfiles, compile your whole world ahead of time, stuff it into a zipfile, and install / distribute that. But for the automatic writing of bytecode files as a side effect of importing the source code, please let the filesystem do i

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Neil Schemenauer
Nick Coghlan wrote: > Henning von Bargen wrote: >> The solution is so obvious: >> >> Why not use a .pyr file that is internally a zip file? I think a Zip file might be the right approach too. Either you could have directories in the zip file for each version, e.g. 2.7/foo.pyc 3.3/foo.p

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Sebastian Rittau
On Sun, Jan 31, 2010 at 12:44:33PM +0100, Georg Brandl wrote: > At least to me, this does not explain why an "unwanted" (why unwanted? If > it's unwanted, set PYTHONDONTWRITEBYTECODE=1) directory is worse than an > "unwanted" file. A directory "feels" different than. For example, typing "ls" in m

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-02 Thread Larry Hastings
On Sun, Jan 31, 2010 at 1:03 PM, Simon Cross wrote: I don't know whether I in favour of using a single pyr folder or not but if a single folder is used I'd definitely prefer the folder to be called __pyr__ rather than .pyr. Guido van Rossum wrote: Exactly what I would prefer. I worry that ha

[Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Pablo Mouzo
On Mon, Feb 1, 2010 at 11:56 PM, Pablo Mouzo wrote: > On Mon, Feb 1, 2010 at 10:23 PM, Paul Du Bois wrote: > [...] >> Sorry, I'm guilty of having assumed that the POSIX API has an >> operation analogous to win32 CreateFile(GENERIC_WRITE, 0 /* ie, >> "FILE_SHARE_NONE"*/). >> >> If shared-reader/si

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Paul Du Bois
>> The python I use (win32 2.6.2) does not complain if it cannot read >> from or write to a .pyc; and thus it handles multiple python processes >> trying to create .pyc files at the same time. Is the .zip case really >> any different? [ snip discussion of difficulty of writing a sharing-safe updat

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Martin v. Löwis
> The python I use (win32 2.6.2) does not complain if it cannot read > from or write to a .pyc; and thus it handles multiple python processes > trying to create .pyc files at the same time. Is the .zip case really > any different? Since .pyc files are an optimization, it seems natural > and correct

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Paul Du Bois
> On Mon, Feb 1, 2010 at 13:19, "Martin v. Löwis" wrote: >> How do you write to a zipfile while others are reading it? On Mon, Feb 1, 2010 at 1:23 PM, Brett Cannon wrote: > By hating concurrency (i.e. I don't have an answer which kills my idea). The python I use (win32 2.6.2) does not complain

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Brett Cannon
On Mon, Feb 1, 2010 at 13:19, "Martin v. Löwis" wrote: >> And I disagree this would be difficult as the PEP suggests given the >> proper file format. For zip files zipimport already has the read code >> in C; it just would require the code to write to a zip file. And as >> for the format I mention

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Martin v. Löwis
> And I disagree this would be difficult as the PEP suggests given the > proper file format. For zip files zipimport already has the read code > in C; it just would require the code to write to a zip file. And as > for the format I mentioned above, that's dead-simple to implement. How do you write

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Antoine Pitrou
Le Mon, 01 Feb 2010 11:35:19 -0800, Brett Cannon a écrit : > > As others have said, an uncompressed zip file could work here. Or even a > file format where the first 4 bytes is the timestamp and then after that > are chunks of length-of-bytecode|magic|bytecode. That allows for opening > a file in

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Brett Cannon
On Sun, Jan 31, 2010 at 11:04, Raymond Hettinger wrote: > > On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: >> Abstract >> >> >> This PEP describes an extension to Python's import mechanism which >> improves sharing of Python source code files among multiple installed >> different versio

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Brett Cannon
On Sun, Jan 31, 2010 at 11:16, Guido van Rossum wrote: > Whoa. This thread already exploded. I'm picking this message to > respond to because it reflects my own view after reading the PEP. > > On Sun, Jan 31, 2010 at 4:13 AM, Hanno Schlichting wrote: >> On Sun, Jan 31, 2010 at 1:03 PM, Simon Cros

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread M.-A. Lemburg
Raymond Hettinger wrote: > > On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: >> Abstract >> >> >> This PEP describes an extension to Python's import mechanism which >> improves sharing of Python source code files among multiple installed >> different versions of the Python interpreter. >

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Antoine Pitrou
> Would you still be a -1 on making it the new scheme the default if it > used a single cache directory instead? That would actually be cleaner > than the current solution rather than messier. Well, I guess no, although additional directories are always more intrusive than additional files (visua

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Gertjan Klein
Hanno Schlichting wrote: >+1 for a single strategy that is used in all cases. The current >solution could be phased out across multiple releases, but in the end >there should be a single approach and no flag. Otherwise some code and >tools will only support one of the approaches, especially if thi

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Martin v. Löwis
> 3. In each top level directory on sys.path, shadow file heirarchy > Major Pro: trivial to separate out all cached files > Major Con: ??? (I got nuthin') The major con of this option (and option 2) is an ambiguity of where to look for in case of packages. In particular for namespace packages

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Ben Finney
Nick Coghlan writes: > Would you still be a -1 on making it the new scheme the default if it > used a single cache directory instead? That would actually be cleaner > than the current solution rather than messier. +0 on a default of “store compiled bytecode files in a single cache directory”. It

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Martin v. Löwis
> But I don't understand how this answers the question. If the > python26-zope.sendmail package doesn't run setup.py, then a > python-zope.sendmail package where you specify at install time which > directory to install the files to isn't going to run setup.py, either. > If the only difference betw

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Silke von Bargen wrote: > >> That still leaves the question of what to do with __file__ (for which >> even the solution in the PEP isn't particularly clean). Perhaps the >> thing to do there is to have __file__ always point to the source file >> and introduce a __file_cached__ that points to the b

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Martin v. Löwis wrote: >> Exactly. How would you define where the pyr folder goes? At the root >> of a package? What if I delete the __init__.py file there? Will the >> existing pyr folder be orphaned and a new one created in each >> subfolder? Unlike VCS working copies, the package / module / scri

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Nick Coghlan
Antoine Pitrou wrote: > Le Sat, 30 Jan 2010 21:04:14 -0800, Jeffrey Yasskin a écrit : >> I have a couple bikesheddy or "why didn't you do this" comments. I'll be >> perfectly satisfied with an answer or a line in the pep. >> >> 1. Why the -R flag? It seems like this is a uniform improvement, so it

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread R. David Murray
On Sun, 31 Jan 2010 19:48:19 +0100, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: > > By the way, the part that caused me the most confusion in the language > > in the PEP was the emphasized *and their dependencies*, as if a package > > having dependencies somehow turned the problem into a fa

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Terry Reedy
On 1/31/2010 4:26 PM, Tim Delaney wrote: The pyc/pyo files are just an optimisation detail, and are essentially temporary. The .pycs for /Lib and similar are*not* temporarily in the sense you are using. They are effectively permanent for as long as the version is installed. They should *n

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Curt Hagenlocher
On Sun, Jan 31, 2010 at 11:16 AM, Terry Reedy wrote: > > 'pycache' would be pretty clear. > Heh -- without the underscores, I read this as "pyc ache". Seems appropriate. -- Curt Hagenlocher c...@hagenlocher.org ___ Python-Dev mailing list Python-Dev@py

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-01-31 Thread Scott Dial
On 1/31/2010 2:04 PM, Raymond Hettinger wrote: > On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: >> It does this by >> allowing many different byte compilation files (.pyc files) to be >> co-located with the Python source file (.py file). > > It would be nice if all the compilation files could

  1   2   >