Re: [Python-Dev] subprocess docs patch

2010-02-01 Thread Chris Rebert
On Mon, Feb 1, 2010 at 12:14 AM, "Martin v. Löwis" wrote: >> Any help you could provide would be appreciated. > > Please use unified diffs in the future. Duly noted. > I'm -0 on this patch; it still has the negative, cautionary-patronizing > tone ("Do not", "can be tricky", "be mindful"), Thank

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-01 Thread Cesare Di Mauro
2010/2/1 Collin Winter > I believe these VMs would have little overlap. I cannot imagine that > Unladen Swallow's needs have much in common with Stackless's, or with > those of a hypothetical register machine to replace the current stack > machine. > > Let's consider that last example in more det

[Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Pablo Mouzo
On Mon, Feb 1, 2010 at 11:56 PM, Pablo Mouzo wrote: > On Mon, Feb 1, 2010 at 10:23 PM, Paul Du Bois wrote: > [...] >> Sorry, I'm guilty of having assumed that the POSIX API has an >> operation analogous to win32 CreateFile(GENERIC_WRITE, 0 /* ie, >> "FILE_SHARE_NONE"*/). >> >> If shared-reader/si

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Antoine Pitrou wrote: > Tres Seaver palladion.com> writes: >> Yup, but that's true for *any* POSIXy envirnnment, not just Python. The >> only sane non-exec mixture is to have a single-thread parent fork, and >> restrict spawning threads to the childr

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Paul Du Bois
>> The python I use (win32 2.6.2) does not complain if it cannot read >> from or write to a .pyc; and thus it handles multiple python processes >> trying to create .pyc files at the same time. Is the .zip case really >> any different? [ snip discussion of difficulty of writing a sharing-safe updat

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Feb 1, 2010, at 6:25 PM, Michael Foord wrote: On 01/02/2010 23:03, Reid Kleckner wrote: On Mon, Feb 1, 2010 at 5:48 PM, Jesse Noller wrote: Your reasonable argument is making it difficult for me to be irrational about this. No problem. :) This begs the question - assuming a

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Antoine Pitrou
Tres Seaver palladion.com> writes: > > Yup, but that's true for *any* POSIXy envirnnment, not just Python. The > only sane non-exec mixture is to have a single-thread parent fork, and > restrict spawning threads to the children. The problem is that we're advocating multiprocessing as the soluti

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Michael Foord
On 01/02/2010 23:03, Reid Kleckner wrote: On Mon, Feb 1, 2010 at 5:48 PM, Jesse Noller wrote: Your reasonable argument is making it difficult for me to be irrational about this. No problem. :) This begs the question - assuming a patch that clones the behavior of win32 for mult

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Antoine Pitrou
Le lundi 01 février 2010 à 23:58 +0100, "Martin v. Löwis" a écrit : > > Interestingly, the POSIX pthread_atfork documentation defines how you > are supposed to do that: create an atfork handler set, and acquire all > mutexes in the prepare handler. Then fork, and have the parent and child > handle

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Reid Kleckner wrote: > On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller wrote: >> I don't disagree there; but then again, I haven't seen this issue >> arise (in my own code)/no bug reports/no test cases that show this to >> be a consistent issue. I'm perf

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Reid Kleckner
On Mon, Feb 1, 2010 at 5:48 PM, Jesse Noller wrote: > Your reasonable argument is making it difficult for me to be irrational > about this. No problem. :) > This begs the question - assuming a patch that clones the behavior of win32 > for multiprocessing, would the default continue to be forkin

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Martin v. Löwis
> We must distinguish between locks owned by the thread which survived the > fork(), and locks owned by other threads. I guess it's possible if we > keep track of the thread id which acquired the lock, and if we give > _whatever_after_fork() the thread id of the thread which initiated the > fork()

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Reid Kleckner
On Mon, Feb 1, 2010 at 5:20 PM, "Martin v. Löwis" wrote: > Instead, we should aim to make Python fork-safe. If the primary concern > is that locks get inherited, we should change the Python locks so that > they get auto-released on fork (unless otherwise specified on lock > creation). This may sou

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Martin v. Löwis
>> Instead, we should aim to make Python fork-safe. If the primary concern >> is that locks get inherited, we should change the Python locks so that >> they get auto-released on fork (unless otherwise specified on lock >> creation). This may sound like an uphill battle, but if there was a >> smart

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Feb 1, 2010, at 5:35 PM, Reid Kleckner wrote: On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller wrote: I don't disagree there; but then again, I haven't seen this issue arise (in my own code)/no bug reports/no test cases that show this to be a consistent issue. I'm perfectly OK with being wr

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Reid Kleckner
On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller wrote: > I don't disagree there; but then again, I haven't seen this issue > arise (in my own code)/no bug reports/no test cases that show this to > be a consistent issue. I'm perfectly OK with being wrong, I'm just > leery to tearing out the internals

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Antoine Pitrou
Le lundi 01 février 2010 à 23:20 +0100, "Martin v. Löwis" a écrit : > > I don't know what spawnl is supposed to do, but it really sounds like > the wrong solution. As far as I understand, I think the idea is to use the same mechanism as under Windows: spawn a new Python interpreter (in a separate

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Martin v. Löwis
> The python I use (win32 2.6.2) does not complain if it cannot read > from or write to a .pyc; and thus it handles multiple python processes > trying to create .pyc files at the same time. Is the .zip case really > any different? Since .pyc files are an optimization, it seems natural > and correct

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Mon, Feb 1, 2010 at 5:20 PM, "Martin v. Löwis" wrote: > Antoine Pitrou wrote: >> Jesse Noller gmail.com> writes: >>> I don't see the need for the change from fork as of yet (for >>> multiprocessing) and I am leery to change the internal implementation >>> and semantics right now, or anytime so

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Martin v. Löwis
Antoine Pitrou wrote: > Jesse Noller gmail.com> writes: >> I don't see the need for the change from fork as of yet (for >> multiprocessing) and I am leery to change the internal implementation >> and semantics right now, or anytime soon. I'd be interested in seeing >> the patch, but if the concern

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Mon, Feb 1, 2010 at 5:08 PM, Antoine Pitrou wrote: >> I don't see spawnl as a viable alternative to fork. I imagine that I, >> and others successfully mix threads and multiprocessing on non-win32 >> platforms just fine, knowing of course that fork() can cause heartburn >> if you have global lo

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Antoine Pitrou
Jesse Noller gmail.com> writes: > > I don't see spawnl as a viable alternative to fork. I imagine that I, > and others successfully mix threads and multiprocessing on non-win32 > platforms just fine, knowing of course that fork() can cause heartburn > if you have global locks code within the fork

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Paul Du Bois
> On Mon, Feb 1, 2010 at 13:19, "Martin v. Löwis" wrote: >> How do you write to a zipfile while others are reading it? On Mon, Feb 1, 2010 at 1:23 PM, Brett Cannon wrote: > By hating concurrency (i.e. I don't have an answer which kills my idea). The python I use (win32 2.6.2) does not complain

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Mon, Feb 1, 2010 at 4:32 PM, Antoine Pitrou wrote: > Jesse Noller gmail.com> writes: >> >> I don't see the need for the change from fork as of yet (for >> multiprocessing) and I am leery to change the internal implementation >> and semantics right now, or anytime soon. I'd be interested in see

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Antoine Pitrou
Jesse Noller gmail.com> writes: > > I don't see the need for the change from fork as of yet (for > multiprocessing) and I am leery to change the internal implementation > and semantics right now, or anytime soon. I'd be interested in seeing > the patch, but if the concern is that global threading

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Brett Cannon
On Mon, Feb 1, 2010 at 13:19, "Martin v. Löwis" wrote: >> And I disagree this would be difficult as the PEP suggests given the >> proper file format. For zip files zipimport already has the read code >> in C; it just would require the code to write to a zip file. And as >> for the format I mention

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Mon, Feb 1, 2010 at 3:09 PM, Pascal Chambon wrote: > > So, if a patch was proposed for the multiprocessing, allowing an unified > "spawnl", thread-safe, semantic, do you think something could prevent its > integration ? > > We may ignore the subprocess module, since fork+exec shouldn't be bothe

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Martin v. Löwis
> And I disagree this would be difficult as the PEP suggests given the > proper file format. For zip files zipimport already has the read code > in C; it just would require the code to write to a zip file. And as > for the format I mentioned above, that's dead-simple to implement. How do you write

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-01 Thread Terry Reedy
On 2/1/2010 1:32 PM, Collin Winter wrote: Hey MA, On Mon, Feb 1, 2010 at 9:58 AM, M.-A. Lemburg wrote: BTW: Some years ago we discussed the idea of pluggable VMs for Python. Wouldn't U-S be a good motivation to revisit this idea ? We could then have a VM based on byte code using a stack machi

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Pascal Chambon
So, if a patch was proposed for the multiprocessing, allowing an unified "spawnl", thread-safe, semantic, do you think something could prevent its integration ? We may ignore the subprocess module, since fork+exec shouldn't be bothered by the (potentially disastrous) state of child process d

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Antoine Pitrou
Le Mon, 01 Feb 2010 11:35:19 -0800, Brett Cannon a écrit : > > As others have said, an uncompressed zip file could work here. Or even a > file format where the first 4 bytes is the timestamp and then after that > are chunks of length-of-bytecode|magic|bytecode. That allows for opening > a file in

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-01 Thread Collin Winter
On Mon, Feb 1, 2010 at 11:17 AM, M.-A. Lemburg wrote: > Collin Winter wrote: >> I think this idea underestimates a) how deeply the current CPython VM >> is intertwined with the rest of the implementation, and b) the nature >> of the changes required by these separate VMs. For example, Unladen >> S

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Brett Cannon
On Sun, Jan 31, 2010 at 11:04, Raymond Hettinger wrote: > > On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: >> Abstract >> >> >> This PEP describes an extension to Python's import mechanism which >> improves sharing of Python source code files among multiple installed >> different versio

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-01 Thread M.-A. Lemburg
Collin Winter wrote: > Hey MA, > > On Mon, Feb 1, 2010 at 9:58 AM, M.-A. Lemburg wrote: >> BTW: Some years ago we discussed the idea of pluggable VMs for >> Python. Wouldn't U-S be a good motivation to revisit this idea ? >> >> We could then have a VM based on byte code using a stack >> machines,

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Brett Cannon
On Sun, Jan 31, 2010 at 11:16, Guido van Rossum wrote: > Whoa. This thread already exploded. I'm picking this message to > respond to because it reflects my own view after reading the PEP. > > On Sun, Jan 31, 2010 at 4:13 AM, Hanno Schlichting wrote: >> On Sun, Jan 31, 2010 at 1:03 PM, Simon Cros

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-01 Thread Collin Winter
Hey MA, On Mon, Feb 1, 2010 at 9:58 AM, M.-A. Lemburg wrote: > BTW: Some years ago we discussed the idea of pluggable VMs for > Python. Wouldn't U-S be a good motivation to revisit this idea ? > > We could then have a VM based on byte code using a stack > machines, one based on word code using a

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython

2010-02-01 Thread M.-A. Lemburg
M.-A. Lemburg wrote: > Collin Winter wrote: >> I added startup benchmarks for Mercurial and Bazaar yesterday >> (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so we >> can use them as more macro-ish benchmarks, rather than merely starting >> the CPython binary over and over again.

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread M.-A. Lemburg
Raymond Hettinger wrote: > > On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: >> Abstract >> >> >> This PEP describes an extension to Python's import mechanism which >> improves sharing of Python source code files among multiple installed >> different versions of the Python interpreter. >

Re: [Python-Dev] Improved Traceback Module

2010-02-01 Thread Benjamin Schweizer
On Fri, Jan 29, 2010 at 08:02:58PM -0500, P.J. Eby wrote: > At 01:24 AM 1/30/2010 +0100, Ludvig Ericson wrote: > >> On 28 jan 2010, at 22:47, P.J. Eby wrote: >> >> > At 07:47 PM 1/28/2010 +0100, Benjamin Schweizer wrote: >> >> >> >> I like the idea of configuring the list of variables with using a

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Antoine Pitrou
> Would you still be a -1 on making it the new scheme the default if it > used a single cache directory instead? That would actually be cleaner > than the current solution rather than messier. Well, I guess no, although additional directories are always more intrusive than additional files (visua

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Gertjan Klein
Hanno Schlichting wrote: >+1 for a single strategy that is used in all cases. The current >solution could be phased out across multiple releases, but in the end >there should be a single approach and no flag. Otherwise some code and >tools will only support one of the approaches, especially if thi

Re: [Python-Dev] subprocess docs patch

2010-02-01 Thread Martin v. Löwis
> Any help you could provide would be appreciated. Please use unified diffs in the future. I'm -0 on this patch; it still has the negative, cautionary-patronizing tone ("Do not", "can be tricky", "be mindful"), as if readers are unable to grasp the description that they just read (and indeed, in

Re: [Python-Dev] PEP 3147: PYC Repository Directories

2010-02-01 Thread Martin v. Löwis
> 3. In each top level directory on sys.path, shadow file heirarchy > Major Pro: trivial to separate out all cached files > Major Con: ??? (I got nuthin') The major con of this option (and option 2) is an ambiguity of where to look for in case of packages. In particular for namespace packages