Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-09 Thread Pascal Chambon
Hello Some update about the spawnl() thingy ; I've adapted the win32 code to have a new unix Popen object, which works with a spawn() semantic. It's quite straightforward, and the mutiprocessing call of a python functions works OK. But I've run into some trouble : synchronization primitives.

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-04 Thread exarkun
On 04:58 pm, jaeda...@gmail.com wrote: Jesse Noller gmail.com> writes: We already have an implementation that spawns a subprocess and then pushes the required state to the child. The fundamental need for things to be pickleable *all the time* kinda makes it annoying to work with. This require

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-04 Thread Ben Walker
Pascal Chambon writes: > I don't really get it there... it seems to me that multiprocessing only > requires picklability for the objects it needs to transfer, i.e those > given as arguments to the called function, and thsoe put into > multiprocessing queues/pipes. Global program data needn't be pic

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-04 Thread Pascal Chambon
Matt Knox a écrit : Jesse Noller gmail.com> writes: We already have an implementation that spawns a subprocess and then pushes the required state to the child. The fundamental need for things to be pickleable *all the time* kinda makes it annoying to work with. just a lurker here...

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-03 Thread Matt Knox
Jesse Noller gmail.com> writes: > We already have an implementation that spawns a > subprocess and then pushes the required state to the child. The > fundamental need for things to be pickleable *all the time* kinda > makes it annoying to work with. > just a lurker here... but this topic hits h

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Martin v. Löwis
> Although I would be in favor of an atfork callback registration system > (similar to atexit), it seems there is no way to solve the fork() > problem automatically with this. Any attempt to acquire/release locks > automatically will lead to deadlocks, as it is necessary to know the > exact program

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Martin v. Löwis
> Le lundi 01 février 2010 à 23:58 +0100, "Martin v. Löwis" a écrit : >> Interestingly, the POSIX pthread_atfork documentation defines how you >> are supposed to do that: create an atfork handler set, and acquire all >> mutexes in the prepare handler. Then fork, and have the parent and child >> han

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Nir Aides
Seems the problem under discussion is already taken care of in Python. Possibly remains to verify that the logic described below does not possibly generate deadlocks. >From the Python docs: http://docs.python.org/c-api/init.html "Another important thing to note about threads is their behaviour in

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Jesse Noller
On Tue, Feb 2, 2010 at 10:34 AM, Pascal Chambon wrote: > > > > The word "dogma" is a good one in this context however. "We" ( ;-)) have > accepted and promoted the dogma that multiprocessing is the solution to > parallelism in the face of the GIL. While it needn't be applicable in any > and > ever

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Pascal Chambon
The word "dogma" is a good one in this context however. "We" ( ;-)) have accepted and promoted the dogma that multiprocessing is the solution to parallelism in the face of the GIL. While it needn't be applicable in any and every situation, we should make it so that it is applicable often enou

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 s...@pobox.com wrote: > Tres> Some applications may seem to work when violating this rule, but > Tres> their developers are doomed to hair loss over time. > > Then for us bald guys it should be okay, right? ;-) Sure: you might even grow hair

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread skip
Tres> Some applications may seem to work when violating this rule, but Tres> their developers are doomed to hair loss over time. Then for us bald guys it should be okay, right? ;-) Skip ___ Python-Dev mailing list Python-Dev@python.org http://m

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Antoine Pitrou wrote: > Tres Seaver palladion.com> writes: >> Note that the "we" in your sentence is not anything like the "quod >> semper quod ubique quod ab omnibus" criterion for accepting dogma: >> mutliprocessing is a tool, and needs to be used a

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Antoine Pitrou
Tres Seaver palladion.com> writes: > > Note that the "we" in your sentence is not anything like the "quod > semper quod ubique quod ab omnibus" criterion for accepting dogma: > mutliprocessing is a tool, and needs to be used according to its nature, > just as with threading. I don't remember eno

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Antoine Pitrou
pobox.com> writes: > > I don't understand. On Unix-y systems isn't spawn* layered on top of > fork/exec? The point is that exec() relinquishes all the existing resources, so the initial fork() becomes an implementation detail (IIUC). > If you were going to > combine both threading and multipro

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread skip
>> I guess spawnl semantic (i.e, like win32's CreateProcess()) can't >> become the default multiprocessing behaviour... Nick> It would also make it much easier to write cross-platform Nick> multiprocessing code (by always using the non-forking semantics Nick> even on fork-capa

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Nick Coghlan
Pascal Chambon wrote: > I guess spawnl semantic (i.e, like win32's CreateProcess()) can't become > the default multiprocessing behaviour, as too many programs implicitly > rely on the whole sharing of data under unix (and py3k itself is maybe > becoming a little too mature for new compatility break

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-02 Thread Pascal Chambon
Although I would be in favor of an atfork callback registration system (similar to atexit), it seems there is no way to solve the fork() problem automatically with this. Any attempt to acquire/release locks automatically will lead to deadlocks, as it is necessary to know the exact program wor

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Antoine Pitrou wrote: > Tres Seaver palladion.com> writes: >> Yup, but that's true for *any* POSIXy envirnnment, not just Python. The >> only sane non-exec mixture is to have a single-thread parent fork, and >> restrict spawning threads to the childr

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Feb 1, 2010, at 6:25 PM, Michael Foord wrote: On 01/02/2010 23:03, Reid Kleckner wrote: On Mon, Feb 1, 2010 at 5:48 PM, Jesse Noller wrote: Your reasonable argument is making it difficult for me to be irrational about this. No problem. :) This begs the question - assuming a

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Antoine Pitrou
Tres Seaver palladion.com> writes: > > Yup, but that's true for *any* POSIXy envirnnment, not just Python. The > only sane non-exec mixture is to have a single-thread parent fork, and > restrict spawning threads to the children. The problem is that we're advocating multiprocessing as the soluti

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Michael Foord
On 01/02/2010 23:03, Reid Kleckner wrote: On Mon, Feb 1, 2010 at 5:48 PM, Jesse Noller wrote: Your reasonable argument is making it difficult for me to be irrational about this. No problem. :) This begs the question - assuming a patch that clones the behavior of win32 for mult

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Antoine Pitrou
Le lundi 01 février 2010 à 23:58 +0100, "Martin v. Löwis" a écrit : > > Interestingly, the POSIX pthread_atfork documentation defines how you > are supposed to do that: create an atfork handler set, and acquire all > mutexes in the prepare handler. Then fork, and have the parent and child > handle

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Reid Kleckner wrote: > On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller wrote: >> I don't disagree there; but then again, I haven't seen this issue >> arise (in my own code)/no bug reports/no test cases that show this to >> be a consistent issue. I'm perf

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Reid Kleckner
On Mon, Feb 1, 2010 at 5:48 PM, Jesse Noller wrote: > Your reasonable argument is making it difficult for me to be irrational > about this. No problem. :) > This begs the question - assuming a patch that clones the behavior of win32 > for multiprocessing, would the default continue to be forkin

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Martin v. Löwis
> We must distinguish between locks owned by the thread which survived the > fork(), and locks owned by other threads. I guess it's possible if we > keep track of the thread id which acquired the lock, and if we give > _whatever_after_fork() the thread id of the thread which initiated the > fork()

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Reid Kleckner
On Mon, Feb 1, 2010 at 5:20 PM, "Martin v. Löwis" wrote: > Instead, we should aim to make Python fork-safe. If the primary concern > is that locks get inherited, we should change the Python locks so that > they get auto-released on fork (unless otherwise specified on lock > creation). This may sou

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Martin v. Löwis
>> Instead, we should aim to make Python fork-safe. If the primary concern >> is that locks get inherited, we should change the Python locks so that >> they get auto-released on fork (unless otherwise specified on lock >> creation). This may sound like an uphill battle, but if there was a >> smart

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Feb 1, 2010, at 5:35 PM, Reid Kleckner wrote: On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller wrote: I don't disagree there; but then again, I haven't seen this issue arise (in my own code)/no bug reports/no test cases that show this to be a consistent issue. I'm perfectly OK with being wr

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Reid Kleckner
On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller wrote: > I don't disagree there; but then again, I haven't seen this issue > arise (in my own code)/no bug reports/no test cases that show this to > be a consistent issue. I'm perfectly OK with being wrong, I'm just > leery to tearing out the internals

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Antoine Pitrou
Le lundi 01 février 2010 à 23:20 +0100, "Martin v. Löwis" a écrit : > > I don't know what spawnl is supposed to do, but it really sounds like > the wrong solution. As far as I understand, I think the idea is to use the same mechanism as under Windows: spawn a new Python interpreter (in a separate

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Mon, Feb 1, 2010 at 5:20 PM, "Martin v. Löwis" wrote: > Antoine Pitrou wrote: >> Jesse Noller gmail.com> writes: >>> I don't see the need for the change from fork as of yet (for >>> multiprocessing) and I am leery to change the internal implementation >>> and semantics right now, or anytime so

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Martin v. Löwis
Antoine Pitrou wrote: > Jesse Noller gmail.com> writes: >> I don't see the need for the change from fork as of yet (for >> multiprocessing) and I am leery to change the internal implementation >> and semantics right now, or anytime soon. I'd be interested in seeing >> the patch, but if the concern

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Mon, Feb 1, 2010 at 5:08 PM, Antoine Pitrou wrote: >> I don't see spawnl as a viable alternative to fork. I imagine that I, >> and others successfully mix threads and multiprocessing on non-win32 >> platforms just fine, knowing of course that fork() can cause heartburn >> if you have global lo

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Antoine Pitrou
Jesse Noller gmail.com> writes: > > I don't see spawnl as a viable alternative to fork. I imagine that I, > and others successfully mix threads and multiprocessing on non-win32 > platforms just fine, knowing of course that fork() can cause heartburn > if you have global locks code within the fork

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Mon, Feb 1, 2010 at 4:32 PM, Antoine Pitrou wrote: > Jesse Noller gmail.com> writes: >> >> I don't see the need for the change from fork as of yet (for >> multiprocessing) and I am leery to change the internal implementation >> and semantics right now, or anytime soon. I'd be interested in see

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Antoine Pitrou
Jesse Noller gmail.com> writes: > > I don't see the need for the change from fork as of yet (for > multiprocessing) and I am leery to change the internal implementation > and semantics right now, or anytime soon. I'd be interested in seeing > the patch, but if the concern is that global threading

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Jesse Noller
On Mon, Feb 1, 2010 at 3:09 PM, Pascal Chambon wrote: > > So, if a patch was proposed for the multiprocessing, allowing an unified > "spawnl", thread-safe, semantic, do you think something could prevent its > integration ? > > We may ignore the subprocess module, since fork+exec shouldn't be bothe

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-02-01 Thread Pascal Chambon
So, if a patch was proposed for the multiprocessing, allowing an unified "spawnl", thread-safe, semantic, do you think something could prevent its integration ? We may ignore the subprocess module, since fork+exec shouldn't be bothered by the (potentially disastrous) state of child process d

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-01-30 Thread Pascal Chambon
/[...] What dangers do you refer to specifically? Something reproducible? -L / Since it's a race condition issue, it's not easily reproducible with normal libraries - which only take threading locks for small moments. But it can appear if your threads make good use of the threading module. By

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-01-30 Thread Pascal Chambon
/[...] What dangers do you refer to specifically? Something reproducible? -L / Since it's a race condition issue, it's not easily reproducible with normal libraries - which only take threading locks for small moments. But it can appear if your threads make good use of the threading module. By

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-01-30 Thread Henning von Bargen
From: Stefan Behnel To: python-dev@python.org Subject: Re: [Python-Dev] Forking and Multithreading - enemy brothers Message-ID: Content-Type: text/plain; charset=ISO-8859-15 Pascal Chambon, 29.01.2010 22:58: I've just recently realized the huge problems surrounding the mix of multithre

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-01-29 Thread Stefan Behnel
Stefan Behnel, 30.01.2010 07:36: > Pascal Chambon, 29.01.2010 22:58: >> I've just recently realized the huge problems surrounding the mix of >> multithreading and fork() - i.e that only the main thread actually >> survived the fork(), and that process data (in particular, >> synchronization primiti

Re: [Python-Dev] Forking and Multithreading - enemy brothers

2010-01-29 Thread Stefan Behnel
Pascal Chambon, 29.01.2010 22:58: > I've just recently realized the huge problems surrounding the mix of > multithreading and fork() - i.e that only the main thread actually > survived the fork(), and that process data (in particular, > synchronization primitives) could be left in a dangerously bro