Hello
Some update about the spawnl() thingy ;
I've adapted the win32 code to have a new unix Popen object, which works
with a spawn() semantic. It's quite straightforward, and the
mutiprocessing call of a python functions works OK.
But I've run into some trouble : synchronization primitives.
On 04:58 pm, jaeda...@gmail.com wrote:
Jesse Noller gmail.com> writes:
We already have an implementation that spawns a
subprocess and then pushes the required state to the child. The
fundamental need for things to be pickleable *all the time* kinda
makes it annoying to work with.
This require
Pascal Chambon writes:
> I don't really get it there... it seems to me that multiprocessing only
> requires picklability for the objects it needs to transfer, i.e those
> given as arguments to the called function, and thsoe put into
> multiprocessing queues/pipes. Global program data needn't be pic
Matt Knox a écrit :
Jesse Noller gmail.com> writes:
We already have an implementation that spawns a
subprocess and then pushes the required state to the child. The
fundamental need for things to be pickleable *all the time* kinda
makes it annoying to work with.
just a lurker here...
Jesse Noller gmail.com> writes:
> We already have an implementation that spawns a
> subprocess and then pushes the required state to the child. The
> fundamental need for things to be pickleable *all the time* kinda
> makes it annoying to work with.
>
just a lurker here... but this topic hits h
> Although I would be in favor of an atfork callback registration system
> (similar to atexit), it seems there is no way to solve the fork()
> problem automatically with this. Any attempt to acquire/release locks
> automatically will lead to deadlocks, as it is necessary to know the
> exact program
> Le lundi 01 février 2010 à 23:58 +0100, "Martin v. Löwis" a écrit :
>> Interestingly, the POSIX pthread_atfork documentation defines how you
>> are supposed to do that: create an atfork handler set, and acquire all
>> mutexes in the prepare handler. Then fork, and have the parent and child
>> han
Seems the problem under discussion is already taken care of in Python.
Possibly remains to verify that the logic described below does not possibly
generate deadlocks.
>From the Python docs: http://docs.python.org/c-api/init.html
"Another important thing to note about threads is their behaviour in
On Tue, Feb 2, 2010 at 10:34 AM, Pascal Chambon
wrote:
>
>
>
> The word "dogma" is a good one in this context however. "We" ( ;-)) have
> accepted and promoted the dogma that multiprocessing is the solution to
> parallelism in the face of the GIL. While it needn't be applicable in any
> and
> ever
The word "dogma" is a good one in this context however. "We" ( ;-)) have
accepted and promoted the dogma that multiprocessing is the solution to
parallelism in the face of the GIL. While it needn't be applicable in any and
every situation, we should make it so that it is applicable often enou
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
s...@pobox.com wrote:
> Tres> Some applications may seem to work when violating this rule, but
> Tres> their developers are doomed to hair loss over time.
>
> Then for us bald guys it should be okay, right? ;-)
Sure: you might even grow hair
Tres> Some applications may seem to work when violating this rule, but
Tres> their developers are doomed to hair loss over time.
Then for us bald guys it should be okay, right? ;-)
Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://m
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Antoine Pitrou wrote:
> Tres Seaver palladion.com> writes:
>> Note that the "we" in your sentence is not anything like the "quod
>> semper quod ubique quod ab omnibus" criterion for accepting dogma:
>> mutliprocessing is a tool, and needs to be used a
Tres Seaver palladion.com> writes:
>
> Note that the "we" in your sentence is not anything like the "quod
> semper quod ubique quod ab omnibus" criterion for accepting dogma:
> mutliprocessing is a tool, and needs to be used according to its nature,
> just as with threading.
I don't remember eno
pobox.com> writes:
>
> I don't understand. On Unix-y systems isn't spawn* layered on top of
> fork/exec?
The point is that exec() relinquishes all the existing resources, so the initial
fork() becomes an implementation detail (IIUC).
> If you were going to
> combine both threading and multipro
>> I guess spawnl semantic (i.e, like win32's CreateProcess()) can't
>> become the default multiprocessing behaviour...
Nick> It would also make it much easier to write cross-platform
Nick> multiprocessing code (by always using the non-forking semantics
Nick> even on fork-capa
Pascal Chambon wrote:
> I guess spawnl semantic (i.e, like win32's CreateProcess()) can't become
> the default multiprocessing behaviour, as too many programs implicitly
> rely on the whole sharing of data under unix (and py3k itself is maybe
> becoming a little too mature for new compatility break
Although I would be in favor of an atfork callback registration system
(similar to atexit), it seems there is no way to solve the fork()
problem automatically with this. Any attempt to acquire/release locks
automatically will lead to deadlocks, as it is necessary to know the
exact program wor
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Antoine Pitrou wrote:
> Tres Seaver palladion.com> writes:
>> Yup, but that's true for *any* POSIXy envirnnment, not just Python. The
>> only sane non-exec mixture is to have a single-thread parent fork, and
>> restrict spawning threads to the childr
On Feb 1, 2010, at 6:25 PM, Michael Foord
wrote:
On 01/02/2010 23:03, Reid Kleckner wrote:
On Mon, Feb 1, 2010 at 5:48 PM, Jesse Noller
wrote:
Your reasonable argument is making it difficult for me to be
irrational
about this.
No problem. :)
This begs the question - assuming a
Tres Seaver palladion.com> writes:
>
> Yup, but that's true for *any* POSIXy envirnnment, not just Python. The
> only sane non-exec mixture is to have a single-thread parent fork, and
> restrict spawning threads to the children.
The problem is that we're advocating multiprocessing as the soluti
On 01/02/2010 23:03, Reid Kleckner wrote:
On Mon, Feb 1, 2010 at 5:48 PM, Jesse Noller wrote:
Your reasonable argument is making it difficult for me to be irrational
about this.
No problem. :)
This begs the question - assuming a patch that clones the behavior of win32
for mult
Le lundi 01 février 2010 à 23:58 +0100, "Martin v. Löwis" a écrit :
>
> Interestingly, the POSIX pthread_atfork documentation defines how you
> are supposed to do that: create an atfork handler set, and acquire all
> mutexes in the prepare handler. Then fork, and have the parent and child
> handle
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Reid Kleckner wrote:
> On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller wrote:
>> I don't disagree there; but then again, I haven't seen this issue
>> arise (in my own code)/no bug reports/no test cases that show this to
>> be a consistent issue. I'm perf
On Mon, Feb 1, 2010 at 5:48 PM, Jesse Noller wrote:
> Your reasonable argument is making it difficult for me to be irrational
> about this.
No problem. :)
> This begs the question - assuming a patch that clones the behavior of win32
> for multiprocessing, would the default continue to be forkin
> We must distinguish between locks owned by the thread which survived the
> fork(), and locks owned by other threads. I guess it's possible if we
> keep track of the thread id which acquired the lock, and if we give
> _whatever_after_fork() the thread id of the thread which initiated the
> fork()
On Mon, Feb 1, 2010 at 5:20 PM, "Martin v. Löwis" wrote:
> Instead, we should aim to make Python fork-safe. If the primary concern
> is that locks get inherited, we should change the Python locks so that
> they get auto-released on fork (unless otherwise specified on lock
> creation). This may sou
>> Instead, we should aim to make Python fork-safe. If the primary concern
>> is that locks get inherited, we should change the Python locks so that
>> they get auto-released on fork (unless otherwise specified on lock
>> creation). This may sound like an uphill battle, but if there was a
>> smart
On Feb 1, 2010, at 5:35 PM, Reid Kleckner wrote:
On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller
wrote:
I don't disagree there; but then again, I haven't seen this issue
arise (in my own code)/no bug reports/no test cases that show this to
be a consistent issue. I'm perfectly OK with being wr
On Mon, Feb 1, 2010 at 5:18 PM, Jesse Noller wrote:
> I don't disagree there; but then again, I haven't seen this issue
> arise (in my own code)/no bug reports/no test cases that show this to
> be a consistent issue. I'm perfectly OK with being wrong, I'm just
> leery to tearing out the internals
Le lundi 01 février 2010 à 23:20 +0100, "Martin v. Löwis" a écrit :
>
> I don't know what spawnl is supposed to do, but it really sounds like
> the wrong solution.
As far as I understand, I think the idea is to use the same mechanism as
under Windows: spawn a new Python interpreter (in a separate
On Mon, Feb 1, 2010 at 5:20 PM, "Martin v. Löwis" wrote:
> Antoine Pitrou wrote:
>> Jesse Noller gmail.com> writes:
>>> I don't see the need for the change from fork as of yet (for
>>> multiprocessing) and I am leery to change the internal implementation
>>> and semantics right now, or anytime so
Antoine Pitrou wrote:
> Jesse Noller gmail.com> writes:
>> I don't see the need for the change from fork as of yet (for
>> multiprocessing) and I am leery to change the internal implementation
>> and semantics right now, or anytime soon. I'd be interested in seeing
>> the patch, but if the concern
On Mon, Feb 1, 2010 at 5:08 PM, Antoine Pitrou wrote:
>> I don't see spawnl as a viable alternative to fork. I imagine that I,
>> and others successfully mix threads and multiprocessing on non-win32
>> platforms just fine, knowing of course that fork() can cause heartburn
>> if you have global lo
Jesse Noller gmail.com> writes:
>
> I don't see spawnl as a viable alternative to fork. I imagine that I,
> and others successfully mix threads and multiprocessing on non-win32
> platforms just fine, knowing of course that fork() can cause heartburn
> if you have global locks code within the fork
On Mon, Feb 1, 2010 at 4:32 PM, Antoine Pitrou wrote:
> Jesse Noller gmail.com> writes:
>>
>> I don't see the need for the change from fork as of yet (for
>> multiprocessing) and I am leery to change the internal implementation
>> and semantics right now, or anytime soon. I'd be interested in see
Jesse Noller gmail.com> writes:
>
> I don't see the need for the change from fork as of yet (for
> multiprocessing) and I am leery to change the internal implementation
> and semantics right now, or anytime soon. I'd be interested in seeing
> the patch, but if the concern is that global threading
On Mon, Feb 1, 2010 at 3:09 PM, Pascal Chambon wrote:
>
> So, if a patch was proposed for the multiprocessing, allowing an unified
> "spawnl", thread-safe, semantic, do you think something could prevent its
> integration ?
>
> We may ignore the subprocess module, since fork+exec shouldn't be bothe
So, if a patch was proposed for the multiprocessing, allowing an unified
"spawnl", thread-safe, semantic, do you think something could prevent
its integration ?
We may ignore the subprocess module, since fork+exec shouldn't be
bothered by the (potentially disastrous) state of child process d
/[...]
What dangers do you refer to specifically? Something reproducible?
-L
/
Since it's a race condition issue, it's not easily reproducible with
normal libraries - which only take threading locks for small moments.
But it can appear if your threads make good use of the threading module.
By
/[...]
What dangers do you refer to specifically? Something reproducible?
-L
/
Since it's a race condition issue, it's not easily reproducible with
normal libraries - which only take threading locks for small moments.
But it can appear if your threads make good use of the threading module.
By
From: Stefan Behnel
To: python-dev@python.org
Subject: Re: [Python-Dev] Forking and Multithreading - enemy brothers
Message-ID:
Content-Type: text/plain; charset=ISO-8859-15
Pascal Chambon, 29.01.2010 22:58:
I've just recently realized the huge problems surrounding the mix of
multithre
Stefan Behnel, 30.01.2010 07:36:
> Pascal Chambon, 29.01.2010 22:58:
>> I've just recently realized the huge problems surrounding the mix of
>> multithreading and fork() - i.e that only the main thread actually
>> survived the fork(), and that process data (in particular,
>> synchronization primiti
Pascal Chambon, 29.01.2010 22:58:
> I've just recently realized the huge problems surrounding the mix of
> multithreading and fork() - i.e that only the main thread actually
> survived the fork(), and that process data (in particular,
> synchronization primitives) could be left in a dangerously bro
44 matches
Mail list logo