[issue6407] multiprocessing Pool should allow custom task queue
Ask Solem added the comment: @swindmill, if you provide a doc/test patch then this can probably be merged. @pitrou, We could change it to `setup_queues`, though I don't think even changing the name of "private" methods is a good idea. It could simply be an alias to `_setup_queues` or vice versa. -- ___ Python tracker <http://bugs.python.org/issue6407> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6288] Update contextlib.nested docs in light of deprecation
Ask Solem added the comment: How would you replace the following functionality with the multiple with statement syntax: x = (A(), B(), C()) with nested(*x) as context: It seems to me that nested() is still useful for this particular use case. -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue6288> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Ask Solem added the comment: I have suspected that this may be necessary, not just merely useful, for some time, and issue6721 seems to verify that. In addition to adding the keyword arg to Process, it should also be added to Pool and Manager. Is anyone working on a patch? If not I will work on a patch asap. -- ___ Python tracker <http://bugs.python.org/issue8713> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: @greg Been very busy lately, just had some time now to look at your patch. I'm very ambivalent about using one SimpleQueue per process. What is the reason for doing that? -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Ask Solem added the comment: This is related to our discussions at #9205 as well (http://bugs.python.org/issue9205), as the final patch there will also fix this issue. -- ___ Python tracker <http://bugs.python.org/issue8296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Ask Solem added the comment: On closer look your patch is also ignoring SystemExit. I think it's beneficial to honor SystemExit, so a user could use this as a means to replace the current process with a new one. If we keep that behavior, the real problem here is that the result handler hangs if the process that reserved a job is gone, which is going to be handled by #9205. Should we mark it as a duplicate? -- ___ Python tracker <http://bugs.python.org/issue8296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: > Does the problem make sense/do you have any ideas for an alternate > solution? Well, I still haven't given up on the trackjobs patch. I changed it to use a single queue for both the acks and the result (see new patch attached: multiprocessing-tr...@82502-termination-trackjobs2.patch) Been running it in production for a few days now, and it seems to work. But the tests still hangs from time to time, it seems they hang more frequent now than in the first patch (this may actually be a good thing:) Would you like to try and identify the cause of this hang? Still haven't been able to. I'm not sure about the overhead of using one queue per process either, but I'm usually running about 8 processes per CPU core for IO bound jobs (adding more processes after that usually doesn't affect performance in positive ways). There's also the overhead of the synchronization (ACK). Not sure if this is important performance-wise, but at least this makes it harder for me to reason about the problem. -- Added file: http://bugs.python.org/file18657/multiprocessing-tr...@82502-termination-trackjobs2.patch ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5573] multiprocessing Pipe poll() and recv() semantics.
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue5573> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3125] test_multiprocessing causes test_ctypes to fail
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue3125> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3111] multiprocessing ppc Debian/ ia64 Ubuntu compilation error
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue3111> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6056] socket.setdefaulttimeout affecting multiprocessing Manager
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue6056> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6362] multiprocessing: handling of errno after signals in sem_acquire()
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue6362> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6407] multiprocessing Pool should allow custom task queue
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue6407> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6417] multiprocessing Process examples: print and getppid
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue6417> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3518] multiprocessing: BaseManager.from_address documented but doesn't exist
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue3518> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6653] Potential memory leak in multiprocessing
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue6653> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue7123> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7060] test_multiprocessing dictionary changed size errors and hang
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue7060> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5862] multiprocessing 'using a remote manager' example errors and possible 'from_address' code leftover
Ask Solem added the comment: Duplicate of 3518? -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue5862> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3735] allow multiple threads to efficiently send the same requests to a processing.Pool without incurring duplicate processing
Ask Solem added the comment: This is a nice feature, but it's also very specific and can be implemented by extending what's already there. Could you make a patch for this that applies to the py3k branch? If no one has the time for this, then we should probably just close the issue, until someone requests it again. -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue3735> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: New patch attach (termination-trackjobs3.patch). > Hmm, a few notes. I have a bunch of nitpicks, but those > can wait for a later iteration. (Just one style nit: I > noticed a few unneeded whitespace changes... please try > not to do that, as it makes the patch harder to read.) Yeah, nitpicks can wait. We need a satisfactory solution first. I forgot about the whitespace, the reason is that the patch was started from the previous trackjobs patch. > - Am I correct that you handle a crashed worker > by aborting all running jobs? No. The job's result is marked with the WorkerLostError, the process is replaced by a new one, and the pool continue to be functional. >- If you're going to the effort of ACKing, why not record >the mapping of tasks to workers so you can be more selective in your >>termination? I does have access to that. There's ApplyResult.worker_pids(). It doesn't terminate anything, it just clean up after whatever terminated. The MapResult could very well discard the job as a whole, but my patch doesn't do that (at least not yet). > Otherwise, what does the ACKing do towards fixing this particular > issue? It's what lets us find out what PID is processing the job. (It also happens to be a required feature to reliably take advantage of external ack semantics (like in AMQP), and also used by my job timeout patch to know when a job was started, and then it shows to be useful in this problem. > - I think in the final version you'd need to introduce some > interthread locking, because otherwise you're going to have weird race > > conditions. I haven't thought too hard about whether you can > get away with just catching unexpected exceptions, but it's > probably better to do the locking. Where is this required? > - I'm getting hangs infrequently enough to make debugging annoying, > and I don't have time to track down the bug right now. Try this: for i in 1 2 3 4 5; ./python.exe test.regrtest -v test_multiprocessing it should show up quickly enough (at least on os x) > Why don't you strip out any changes that are not needed (e.g. AFAICT, > the > ACK logic), make sure there aren't weird race conditions, > and if we start converging on a patch that looks right from a high > level we > can try to make it work on all the corner case? See the updated patch. I can't remove the ACK, but I removed the accept_callback, as it's not strictly needed to solve this problem. -- Added file: http://bugs.python.org/file18664/multiprocessing-tr...@82502-termination-trackjobs3.patch ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: > - A worker removes a job from the queue and is killed before > sending an ACK. Yeah, this may be a problem. I was thinking we could make sure the task is acked before child process shutdown. Kill -9 is then not safe, but do we really want to guarantee that in multiprocessing? In celery we're safe by using AMQP's ack trasnaction anyway. The same could be said if there's a problem with the queue though. Maybe using ack timeouts? We know how many worker processes are free already. > A worker removes a job from the queue, sends an ACK, and then is > killed. Due to bad luck with the scheduler, the parent cleans the > worker before the parent has recorded the worker pid. Guess we need to consume from the result queue until it's empty. > You're now reading from self._cache in one thread but writing it in > another. Yeah, I'm not sure if SimpleQueue is managed by a lock already. Should maybe use a lock if it isn't. > What happens if a worker sends a result and then is killed? In the middle of sending? Or, if not I don't think this matters. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: By the way, I'm also working on writing some simple benchmarks for the multiple queues approach, just to see if theres at all an overhead to worry about. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3735] allow multiple threads to efficiently send the same requests to a processing.Pool without incurring duplicate processing
Changes by Ask Solem : -- resolution: -> postponed stage: unit test needed -> needs patch ___ Python tracker <http://bugs.python.org/issue3735> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4892] Sending Connection-objects over multiprocessing connections fails
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue4892> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3831] Multiprocessing: Expose underlying pipe in queues
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue3831> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5501] Update multiprocessing docs re: freeze_support
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue5501> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8534] multiprocessing not working from egg
Changes by Ask Solem : -- keywords: +needs review nosy: +asksol ___ Python tracker <http://bugs.python.org/issue8534> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3093] Namespace pollution from multiprocessing
Changes by Ask Solem : -- nosy: +asksol stage: -> needs patch ___ Python tracker <http://bugs.python.org/issue3093> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6407] multiprocessing Pool should allow custom task queue
Ask Solem added the comment: are there really any test/doc changes needed for this? -- ___ Python tracker <http://bugs.python.org/issue6407> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6407] multiprocessing Pool should allow custom task queue
Changes by Ask Solem : -- stage: needs patch -> unit test needed ___ Python tracker <http://bugs.python.org/issue6407> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9733] Can't iterate over multiprocessing.managers.DictProxy
Ask Solem added the comment: > I expected I could iterate over a DictProxy as I do over a > regular dict. DictProxy doesn't support iterkeys(), itervalues(), or iteritems() either. So while iter(d) could do iter(d.keys()) behind the scenes, it would mask the fact that this would not return an *iterator* over the keys, but send a potentially long list of keys back to the client. -- ___ Python tracker <http://bugs.python.org/issue9733> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3125] test_multiprocessing causes test_ctypes to fail
Ask Solem added the comment: As no one has been able to confirm that this is still an issue, I'm closing it as "out of date". The issue can be reopened if necessary. -- resolution: accepted -> out of date status: open -> closed ___ Python tracker <http://bugs.python.org/issue3125> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3111] multiprocessing ppc Debian/ ia64 Ubuntu compilation error
Ask Solem added the comment: As no one is able to confirm that this is still an issue, I'm closing it. It can be reopened if necessary. -- resolution: -> out of date ___ Python tracker <http://bugs.python.org/issue3111> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9801] Can not use append/extend to lists in a multiprocessing manager dict
Ask Solem added the comment:
Maybe surprising but not so weird if you think about what happens
behind the scenes.
When you do
>>> x = man.list()
>>> x.append({})
You send an empty dict to the manager to be appended to x
when do:
>>> x[0]
{}
you receive a local copy of the empty dict from the manager process.
So this:
>>> x[0]["a"] = 5
will only modify the local copy.
What you would have to do is:
>>> x.append({})
>>> t = x[0]
>>> t["a"] = 5
>>> x[0] = t
This will not be atomic of course, so this may be something
to take into account.
What maybe could be supported is something like:
>>> x[0] = manager.dict()
>>>x[0]["foo"] = "bar"
but otherwise I wouldn't consider this a bug.
--
nosy: +asksol
___
Python tracker
<http://bugs.python.org/issue9801>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7707] multiprocess.Queue operations during import can lead to deadlocks
Ask Solem added the comment: I created a small doc patch for this (attached). -- keywords: +needs review, patch nosy: +asksol versions: +Python 3.1 -Python 2.6 Added file: http://bugs.python.org/file18967/multiprocessing-issue7707.patch ___ Python tracker <http://bugs.python.org/issue7707> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8028] self.terminate() from a multiprocessing.Process raises AttributeError exception
Ask Solem added the comment: Could you please reduce this to the shorted possible example that reproduces the problem? -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue8028> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8094] Multiprocessing infinite loop
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue8094> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8144] muliprocessing shutdown infinite loop
Ask Solem added the comment: Did you finish the code to reproduce the problem? -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue8144> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10037] multiprocessing.pool processes started by worker handler stops working
New submission from Ask Solem : While working on an "autoscaling" (yes, people call it that...) feature for Celery, I noticed that the processes created by the _handle_workers thread doesn't always work. I have reproduced this in general, by just using the maxtasksperchild feature and letting the workers terminate themselves so this seems to have always been an issue (just not easy to reproduce unless workers are created with some frequency) I'm not quite sure of the reason yet, but I finally managed to track it down to the workers being stuck while receiving from the queue. The patch attached seems to resolve the issue by polling the queue before trying to receive. I know this is short, I may have some more data later. -- components: Library (Lib) files: multiprocessing-worker-poll.patch keywords: needs review, patch messages: 118062 nosy: asksol priority: critical severity: normal stage: patch review status: open title: multiprocessing.pool processes started by worker handler stops working type: behavior versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3 Added file: http://bugs.python.org/file19139/multiprocessing-worker-poll.patch ___ Python tracker <http://bugs.python.org/issue10037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10128] multiprocessing.Pool throws exception with __main__.py
Ask Solem added the comment: Is this on Windows? Does it work for you now? -- ___ Python tracker <http://bugs.python.org/issue10128> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10128] multiprocessing.Pool throws exception with __main__.py
Changes by Ask Solem : -- resolution: -> invalid status: open -> closed ___ Python tracker <http://bugs.python.org/issue10128> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10174] multiprocessing expects sys.stdout to have a fileno/close method.
Ask Solem added the comment: Please add the traceback, I can't seem to find any obvious places where this would happen now. Also, what version are you currently using? I agree with the fileno, but I'd say close is a reasonable method to implement, especially for stdin/stdout/stderr -- ___ Python tracker <http://bugs.python.org/issue10174> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4999] multiprocessing.Queue does not order objects
Ask Solem added the comment: Updated doc patch -- nosy: +asksol Added file: http://bugs.python.org/file19350/issue-4999.diff ___ Python tracker <http://bugs.python.org/issue4999> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Ask Solem added the comment: AFAICS the object will be pickled twice with this patch. See Modules/_multiprocessing/connection.h: connection_send_obj. -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Ask Solem added the comment: aha, no. I see now you use connection.send_bytes instead. Then I can't think of any issues with this patch, but I don't know why it was done this way in the first place. -- ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7200] multiprocessing deadlock on Mac OS X when queue collected before process terminates
Ask Solem added the comment: Queue uses multiprocessing.util.Finalize, which uses weakrefs to track when the object is out of scope, so this is actually expected behavior. IMHO it is not a very good approach, but changing the API to use explicit close methods is a little late at this point, I guess. -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue7200> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6407] multiprocessing Pool should allow custom task queue
Ask Solem added the comment: Matthew, would you be willing to write tests + documentation for this? -- ___ Python tracker <http://bugs.python.org/issue6407> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7474] multiprocessing.managers.SyncManager managed object creation fails when started outside of invoked file
Ask Solem added the comment: I can't seem to reproduce this on trunk... -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue7474> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5573] multiprocessing Pipe poll() and recv() semantics.
Ask Solem added the comment: I don't know about the socket internals, but I find the behavior acceptable. It may not be feasible to change it now anyway, as there may be people already depending on it (e.g. not handling errors occurring at poll) -- ___ Python tracker <http://bugs.python.org/issue5573> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7292] Multiprocessing Joinable race condition?
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue7292> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5930] Transient error in multiprocessing (test_number_of_objects)
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue5930> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9733] Can't iterate over multiprocessing.managers.DictProxy
Changes by Ask Solem : -- resolution: -> invalid status: open -> closed ___ Python tracker <http://bugs.python.org/issue9733> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3876] multiprocessing does not compile on systems which do not define sem_timedwait
Ask Solem added the comment: What is the status of this issue? There are several platform listed here, which I unfortunately don't have access to. -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue3876> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9955] multiprocessing.Pipe segmentation fault when recv of unpicklable object
Ask Solem added the comment: Can't reproduce on Python 2.7, but can indeed reproduce on 2.6. Issue fixed? -- ___ Python tracker <http://bugs.python.org/issue9955> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10133] multiprocessing: conn_recv_string() broken error handling
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue10133> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8028] self.terminate() from a multiprocessing.Process raises AttributeError exception
Ask Solem added the comment: It seems that Process.terminate is not meant to be used by the child, but only the parent. >From the documentation: "Note that the start(), join(), is_alive() and exit_code methods should only be called by the process that created the process object." Either terminate() should be added to this list, or terminate should be patch to call sys.exit in a child process. I vote for the former, so I attached a doc patch. -- keywords: +patch Added file: http://bugs.python.org/file19466/i8028.patch ___ Python tracker <http://bugs.python.org/issue8028> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5573] multiprocessing Pipe poll() and recv() semantics.
Changes by Ask Solem : -- resolution: -> invalid status: open -> closed ___ Python tracker <http://bugs.python.org/issue5573> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Ask Solem added the comment: Pickling on put makes sense to me. I can't think of cases where this could break existing code either. I think this may also resolve issue 8323 -- stage: -> unit test needed ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8028] self.terminate() from a multiprocessing.Process raises AttributeError exception
Ask Solem added the comment: Since you can't specify the return code, `self.terminate` is less flexible than `sys.exit`. I think the original intent is clear here, the method is there for the parent to control the child. You are of course welcome to argue otherwise. By the way, I just read the code and noticed that it handles SystemExit well, and supports using it to set the return code: class X(Process): def run(self): if not frobulating: raise SystemExit(255) -- ___ Python tracker <http://bugs.python.org/issue8028> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10305] Cleanup up ResourceWarnings in multiprocessing
Ask Solem added the comment: ah, this is something I've seen as well, its part of a bug that I haven't created an issue for yet. -- ___ Python tracker <http://bugs.python.org/issue10305> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11743] Rewrite PipeConnection and Connection in pure Python
Ask Solem added the comment: This is great! I always wondered if it was really necessary to use C for this. 10µs overhead should be worth it ;) I've read the patch, but not carefully. So far nothing jumps at me either. Cheers! -- ___ Python tracker <http://bugs.python.org/issue11743> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10673] multiprocess.Process join method - timeout indistinguishable from success
Ask Solem added the comment: While it makes sense for `join` to raise an error on timeout, that could possibly break existing code, so I don't think that is an option. Adding a note in the documentation would be great. -- ___ Python tracker <http://bugs.python.org/issue10673> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36403] AsyncIterator on 3.7: __aiter__ no longer honors finally blocks
Ask Solem added the comment: Ah, so the extra call_soon means it needs a: [code] loop.run_until_complete(asyncio.sleep(0))``` [/code] before the self.assertTrue(it.finally_executed) to finish executing agen.close(). Why is create_task different? Does it execute an iteration of the generator immediately? Seems good for this behavior to be consistent, but not sure how difficult that would be. -- ___ Python tracker <https://bugs.python.org/issue36403> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36403] AsyncIterator on 3.7: __aiter__ no longer honors finally blocks
Ask Solem added the comment: Perhaps we could add a self._finally to the event loop itself? Like loop._ready, but a list of callbacks run_until_complete will call before returning? -- ___ Python tracker <https://bugs.python.org/issue36403> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10037] multiprocessing.pool processes started by worker handler stops working
Ask Solem added the comment: Well, I still don't know exactly why restarting the socket read made it work, but the patch solved an issue where newly started pool processes would be stuck in socket read forever (happening to maybe 1/500 new processes) This and a dozen other pool related fixes are in my billiard fork of multiprocessing, e.g. what you describe in your comment: # trying res.get() would block forever works in billiard, where res.get() will raise WorkerLostError in that case. https://github.com/celery/billiard/ Earlier commit history for the pool can be found in Celery: https://github.com/ask/celery/commits/2.5/celery/concurrency/processes/pool.py My eventual goal is to merge these fixes back into Python, but except for people using Python 3.x, they would have to use billiard for quite some time anyway, so I don't feel in a hurry. I think this issue can be closed, the worker handler is simply borked and we could open up a new issue deciding how to fix it (merging billiard.Pool or someting else). (btw, Richard, you're sbt? I was trying to find your real name to give you credit for the no_execv patch in billiard) -- ___ Python tracker <http://bugs.python.org/issue10037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10037] multiprocessing.pool processes started by worker handler stops working
Ask Solem added the comment: Later works, or just close it. I can open up a new issue to merge the improvements in billiard later. > The execv stuff certainly won't go in by Py3.3. There has not been > consensus that adding it is a good idea. > (I also have the unit tests passing with a "fork server": the server >process > is forked at the beginning of the program and then forked >children of the > server process are started on request. It is about 10 >times faster then > using execv, and almost as fast as simple forking.) Ah, a working 'fork server' would be just as good. Btw, Billiard now supports running Pool without threads, using epoll/kqueue/select instead. So Celery uses that when it can be nonblocking, and execv when it can't. It performs way better without threads, and in addition shutdown + replacing worker processes is much more responsive. Changing the default Pool is not going to happen, but ncluding a simple select() based Pool would be possible, and then it could also easily work with Twisted, Eventlet, Gevent, etc. (especially now that the Connection is rewritten in pure python). -- ___ Python tracker <http://bugs.python.org/issue10037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9592] Limitations in objects returned by multiprocessing Pool
Ask Solem added the comment: I vote to close too as it's very hard to fix in a clean way. A big problem though is that there is a standard for defining exceptions, that also ensures that the exception is pickleable (always call Exception.__init__ with original args), that is not documented (http://docs.python.org/tutorial/errors.html#user-defined-exceptions). Celery has an elaborate mechanism to rewrite unpickleable exceptions, but it's a massive workaround just to keep the workers running, and shouldn't be part of the stdlib. It would help if the Python documentation mentioned this though. Related: http://docs.celeryproject.org/en/latest/userguide/tasks.html#creating-pickleable-exceptions -- ___ Python tracker <http://bugs.python.org/issue9592> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9248] multiprocessing.pool: Proposal: "waitforslot"
Ask Solem added the comment: This patch is quite dated now and I have fixed many bugs since. The feature is available in billiard and is working well but The code has diverged quite a lot from python trunk. I will be updating billiard to reflect the changes for Python 3.4 soon (billiard is currently 3.3). I think we can forget about taking individual patches from billiard for now, and instead maybe merge the codebases at some point if there's interest. we have a version of multiprocessing.Pool using async IO and one pipe per process that drastically improves performance and also avoids the threads+forking issues (well, not the initial fork), but I have not yet adapted it to use the new asyncio module in 3.4 So suggestion is to close this and rather get a discussion going for combining our efforts. -- ___ Python tracker <http://bugs.python.org/issue9248> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5672] Implement a way to change the python process name
Ask Solem added the comment: Amaury Forgeot d'Arc, wrote: And I'd follow the same path: provide a way to build a launcher - a .exe file that simply starts python with the given script. Sounds good, but how would you expect to set the process name for a subprocess, like a multiprocessing.Process? Make one shell script for every process you want to launch and use exec? Maybe even dynamically create the scripts? This is not a solution. -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue5672> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7285] multiprocessing module, example code error
Changes by Ask Solem : Added file: http://bugs.python.org/file15374/7285.patch ___ Python tracker <http://bugs.python.org/issue7285> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5738] multiprocessing example wrong
Ask Solem added the comment: Are we sure this fits the scope of multiprocessing? It's a nice feature, but such a long and complex example in the documentation is wrong IMHO, if this is something people need it should be implemented as a reusable solution, not as an example people copy and paste. -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue5738> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6615] multiprocessing logging support test
Changes by Ask Solem : -- keywords: +patch Added file: http://bugs.python.org/file15375/6615.patch ___ Python tracker <http://bugs.python.org/issue6615> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6064] Add "daemon" argument to threading.Thread constructor
Ask Solem added the comment: Patch for multiprocessing, adding the daemon kwarg attached. -- nosy: +asksol Added file: http://bugs.python.org/file15376/6064-multiprocessing-daemon-kwarg.patch ___ Python tracker <http://bugs.python.org/issue6064> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9162] License for multiprocessing files
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue9162> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Changes by Ask Solem : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: termination.patch, in the result handler you've added: while cache and thread._state != TERMINATE and not failed why are you terminating the second pass after finding a failed process? Unpickleable errors and other errors occurring in the worker body are not exceptional cases, at least not now that the pool is supervised by _handle_workers. I think the result should be set also in this case, so the user can inspect the exception after the fact. I have some other suggestions too, so I will review this patch tomorrow. For shutdown.patch, I thought this only happened in the worker handler, but you've enabled this for the result handler too? I don't care about the worker handler, but with the result handler I'm worried that I don't know what ignoring these exceptions actually means. For example, is there a possibility that we may lose results at shutdown? -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: > Unfortunately, if you've lost a worker, you are no > longer guaranteed that cache will eventually be empty. > In particular, you may have lost a task, which could > result in an ApplyResult waiting forever for a _set call. > More generally, my chief assumption that went into this > is that the unexpected death of a worker process is > unrecoverable. It would be nice to have a better workaround > than just aborting everything, but I couldn't see a way > to do that. It would be a problem if the process simply disappeared, But in this case you have the ability to put a result on the queue, so it doesn't have to wait forever. For processes disappearing (if that can at all happen), we could solve that by storing the jobs a process has accepted (started working on), so if a worker process is lost, we can mark them as failed too. > I could be wrong, but that's not what my experiments > were indicating. In particular, if an unpickleable error occurs, > then a task has been lost, which means that the relevant map, > apply, etc. will wait forever for completion of the lost task. It's lost now, but not if we handle the error... For a single map operation this behavior may make sense, but what about someone running the pool as s long-running service for users to submit map operations to? Errors in this context are expected to happen, even unpickleable errors. I guess that the worker handler works as a supervisor is a side effect, as it was made for the maxtasksperchild feature, but for me it's a welcome one. With the supervisor in place, multiprocessing.pool is already fairly stable to be used for this use case, and there's not much to be done to make it solid (Celery is already running for months without issue, unless there's a pickling error...) > That does sound useful. Although, how can you determine the > job (and the value of i) if it's an unpickleable error? > It would be nice to be able to retrieve job/i without having > to unpickle the rest. I was already working on this issue last week actually, and I managed to do that in a way that works well enough (at least for me): http://github.com/ask/celery/commit/eaa4d5ddc06b000576a21264f11e6004b418bda1#diff-1 -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: I think I misunderstood the purpose of the patch. This is about handling errors on get(), not on put() like I was working on. So sorry for that confusion. What kind of errors are you having that makes the get() call fail? If the queue is not working, then I guess the only sensible approach is to shutdown the pool like suggested. I'll open up another issue for unpickleable errors then. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9244] multiprocessing.pool: Pool crashes if worker can't encode result (with patch)
New submission from Ask Solem : If the target function returns an unpickleable value the worker process crashes. This patch tries to safely handle unpickleable errors, while enabling the user to inspect such errors after the fact. In addition a new argument has been added to apply_async: error_callback. This is an optional callback that is called if the job raises an exception. The signature of the callback is `callback(exc)`. -- components: Library (Lib) files: multiprocessing-tr...@82502-handle_worker_encoding_errors.patch keywords: patch messages: 110173 nosy: asksol, jnoller priority: normal severity: normal status: open title: multiprocessing.pool: Pool crashes if worker can't encode result (with patch) type: behavior versions: Python 2.6, Python 2.7 Added file: http://bugs.python.org/file17982/multiprocessing-tr...@82502-handle_worker_encoding_errors.patch ___ Python tracker <http://bugs.python.org/issue9244> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: For reference I opened up a new issue for the put() case here: http://bugs.python.org/issue9244 -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9244] multiprocessing.pool: Worker crashes if result can't be encoded result (with patch)
Changes by Ask Solem : -- title: multiprocessing.pool: Pool crashes if worker can't encode result (with patch) -> multiprocessing.pool: Worker crashes if result can't be encoded result (with patch) ___ Python tracker <http://bugs.python.org/issue9244> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9244] multiprocessing.pool: Worker crashes if result can't be encoded
Changes by Ask Solem : -- title: multiprocessing.pool: Worker crashes if result can't be encoded result (with patch) -> multiprocessing.pool: Worker crashes if result can't be encoded ___ Python tracker <http://bugs.python.org/issue9244> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9248] multiprocessing.pool: Proposal: "waitforslot"
New submission from Ask Solem : This patch adds the `waitforslot` argument to apply_async. If set to `True`, apply_async will not return until there is a worker available to process the job. This is implemented by a semaphore that is released by the result handler whenever a new result is ready. The semaphore is also released when the supervisor (worker_handler) finds a worker process that has been unexpectedly terminated. This is already in use by Celery 2.0, which ships with its own modified version of multiprocessing.pool. I'm not sure about the name ``waitforslot``, I think I may even hate it, but haven't been able to come up with a better name for it yet. -- components: Library (Lib) messages: 110193 nosy: asksol, jnoller priority: normal severity: normal status: open title: multiprocessing.pool: Proposal: "waitforslot" versions: Python 2.6, Python 2.7 ___ Python tracker <http://bugs.python.org/issue9248> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9248] multiprocessing.pool: Proposal: "waitforslot"
Changes by Ask Solem : -- keywords: +patch Added file: http://bugs.python.org/file17985/multiprocessing-tr...@82502-apply-semaphore.patch ___ Python tracker <http://bugs.python.org/issue9248> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: There's one more thing if exitcode is not None: cleaned = True if exitcode != 0 and not worker._termination_requested: abnormal.append((worker.pid, exitcode)) Instead of restarting crashed worker processes it will simply bring down the pool, right? If so, then I think it's important to decide whether we want to keep the supervisor functionality, and if so decide on a recovery strategy. Some alternatives are: A) Any missing worker brings down the pool. B) Missing workers will be replaced one-by-one. A maximum-restart-frequency decides when the supervisor should give up trying to recover the pool, and crash it. C) Same as B, except that any process crashing when trying to get() will bring down the pool. I think the supervisor is a good addition, so I would very much like to keep it. It's also a step closer to my goal of adding the enhancements added by Celery to multiprocessing.pool. Using C is only a few changes away from this patch, but B would also be possible in combination with my accept_callback patch. It does pose some overhead, so it depends on the level of recovery we want to support. accept_callback: this is a callback that is triggered when the job is reserved by a worker process. The acks are sent to an additional Queue, with an additional thread processing the acks (hence the mentioned overhead). This enables us to keep track of what the worker processes are doing, also get the PID of the worker processing any given job (besides from recovery, potential uses are monitoring and the ability to terminate a job (ApplyResult.terminate?). See http://github.com/ask/celery/blob/master/celery/concurrency/processes/pool.py -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9244] multiprocessing.pool: Worker crashes if result can't be encoded
Ask Solem added the comment:
> To be clear, the errback change and the unpickleable result
> change are actually orthogonal, right?
Yes, it could be a separate issue. Jesse, do you think I should I open
up a separate issue for this?
> Why not add an error_callback for map_async as well?
That's a good idea!
> Any reason you chose to use a different internal name
> (errback versus error_callback)? It seems cleaner to me
> to be consistent about the name.
It was actually a mistake. The argument was ``errback`` before, so
it's just a leftover from the previous name.
> In general, I'm wary of nonessential whitespace changes...
> did you mean to include these?
Of course not.
> Using "assertTrue" seems misleading. "assertIsNotNone" is what
> really mean, right? Although, I believe that's redundant,
> since presumably self.assertIsInstance(None, KeyError) will
> error out anyway (I haven't verified this).
bool(KeyError("foo")) is True and bool(None) is False, so it works either way.
It could theoretically result in a false negative if
the exception class tested overrides __nonzero__, but that is unlikely
to happen as the target always returns KeyError anyway (and the test below
ensures it) It's just a habit of mine, unless I really want to test for
Noneness, I just use assertTrue, but I'm not against changing it to
assertIsNotNone either.
> Under what circumstances would these be None? (Perhaps you
> want wrapped.exc != 'None'?) The initializer for
> MaybeEncodingError enforces the invariant that exc/value are strings
> right?
It's just to test that these are actually set to something.
Even an empty string passes with assertIsNone instead of assertTrue.
Maybe it's better to test the values set, but I didn't bother.
--
___
Python tracker
<http://bugs.python.org/issue9244>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: Jesse wrote, > We can work around the shutdown issue (really, bug 9207) by > ignoring the exception such as shutdown.patch does, or passing in > references/adding references to the functions those methods need. Or (as > Brett suggested) converting them to class methods and adding references to > the class. Or passing them in via the signature like this > _handle_workers(arg, _debug=debug), etc. Greg wrote, > Another option would be to enable only for the worker handler. I > don't have a particularly great sense of what the Right Thing to > do here is. I don't think _make_shutdown_safe should be added to the result handler. If the error can indeed happen there, then we need to solve it in a way that enables it to finish the work. Jesse, how hard is it to fix the worker handler by passing the references? Note that _worker_handler is not important to complete shutdown at this point, but it may be in the future (it seems termination.patch already changes this) -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: Greg, > Before I forget, looks like we also need to deal with the > result from a worker being un-unpickleable: This is what my patch in bug 9244 does... > Yep. Again, as things stand, once you've lost an worker, > you've lost a task, and you can't really do much about it. > I guess that depends on your application though... is your > use-case such that you can lose a task without it mattering? > If tasks are idempotent, one could have the task handler > resubmit them, etc.. But really, thinking about the failure > modes I've seen (OOM kills/user-initiated interrupt) I'm not > sure under what circumstances I'd like the pool to try to > recover. Losing a task is not fun, but there may still be other tasks running that are just as important. I think you're thinking from a map_async perspective here. user-initiated interrupts, this is very important to recover from, think of some badly written library code suddenly raising SystemExit, this shouldn't terminate other jobs, and it's probably easy to recover from, so why shouldn't it try? > The idea of recording the mapping of tasks -> workers > seems interesting. Getting all of the corner cases could > be hard (e.g. making removing a task from the queue and > recording which worker did the removing atomic, detecting if the worker > crashed while still holding the queue lock) and doing > this would require extra mechanism. This feature does seem > to be useful for pools running many different jobs, because > that way a crashed worker need only terminate one job. I think I may have an alternative solution. Instead of keeping track of what the workers are doing, we could simply change the result handler so it gives up when there are no more alive processes. while state != TERMINATE: result = get(timeout=1) if all_processes_dead(): break; -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: Ok. I implemented my suggestions in the patch attached (multiprocessing-tr...@82502-termination2.patch) What do you think? Greg, Maybe we could keep the behavior in termination.patch as an option for map jobs? It is certainly a problem that map jobs won't terminate until the pool is joined. -- Added file: http://bugs.python.org/file18013/multiprocessing-tr...@82502-termination2.patch ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9244] multiprocessing.pool: Worker crashes if result can't be encoded
Ask Solem added the comment: Updated patch with Greg's suggestions. (multiprocessing-tr...@82502-handle_worker_encoding_errors2.patch) -- Added file: http://bugs.python.org/file18014/multiprocessing-tr...@82502-handle_worker_encoding_errors2.patch ___ Python tracker <http://bugs.python.org/issue9244> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Changes by Ask Solem : Removed file: http://bugs.python.org/file18013/multiprocessing-tr...@82502-termination2.patch ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: Just some small cosmetic changes to the patch. (added multiprocessing-tr...@82502-termination3.patch) -- Added file: http://bugs.python.org/file18015/multiprocessing-tr...@82502-termination3.patch ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: > Really? I could be misremembering, but I believe you deal > with the case of the result being unpickleable. I.e. you > deal with the put(result) failing, but not the get() in the > result handler. Your example is demonstrating the pickle error on put(), not on get(). > Does my sample program work with your patch applied? Yeah, check this out: /opt/devel/Python/trunk(master)$> patch -p1 < multiprocessing-tr...@82502-handle_worker_encoding_errors2.patch patching file Lib/multiprocessing/pool.py patching file Lib/test/test_multiprocessing.py /opt/devel/Python/trunk(master)$> ./python.exe Python 2.7 (unknown, Jul 13 2010, 13:28:35) [GCC 4.2.1 (Apple Inc. build 5659)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import multiprocessing >>> def foo(): ... return lambda: 42 ... >>> p = multiprocessing.Pool(2) >>> p.apply_async(foo).get() Traceback (most recent call last): File "", line 1, in File "/opt/devel/Python/trunk/Lib/multiprocessing/pool.py", line 518, in get raise self._value multiprocessing.pool.MaybeEncodingError: Error sending result: ' at 0x1005477d0>'. Reason: 'Can't pickle : attribute lookup __builtin__.function failed' >>> import operator >>> p.apply_async(operator.add, (2, 2)).get() 4 > To be clear, in this case I was thinking of KeyboardInterrupts. In termination2.patch I handle BaseExceptions, by exiting the worker process, and then letting the _worker_handler replace the process. It's very useful, because then people can kill -INT the worker process if they want to cancel the job, and without breaking other jobs running. > From our differing use-cases, I do think it could make sense as > a configuration option, but where it probably belongs is on the > wait() call of ApplyResult. Indeed! This could be done by adding listeners for this type of errors. pool.add_worker_missing_callback(fun) So MapResults could install a callback like this: def __init__(): ... _pool.add_worker_missing_callback(self._on_worker_missing) ... def _on_worker_missing(self): err = WorkerLostError( "Worker lost while running map job") self._set(None, (False, err)) What do you think about that? IMHO, even though the worker lost could be unrelated to the map job in question, it would still be a better alternative than crashing the whole pool. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: > but if you make a blocking call such as in the following program, > you'll get a hang Yeah, and for that we could use the same approach as for the maps. But, I've just implemented the accept callback approach, which should be superior. Maps/Apply fails instantly as soon as a worker process crashes, but the pool remains fully functional. Patch multiprocessing-tr...@82502-termination-trackjobs.patch added. There seems to be some race conditions left, because some of the tests breaks from time to time. Maybe you can pinpoint it before me. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Changes by Ask Solem : Added file: http://bugs.python.org/file18026/multiprocessing-tr...@82502-termination-trackjobs.patch ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: >At first glance, looks like there are a number of sites where you don't >>change the blocking calls to non-blocking calls (e.g. get()). Almost >all of >the get()s have the potential to be called when there is no >possibility for >them to terminate. > >I might recommend referring to my original termination.patch... I >believe I >tracked down the majority of such blocking calls. I thought the EOF errors would take care of that, at least this has been running in production on many platforms without that happening. >In the interest of simplicity though, I'm beginning to think that the >right >answer might be to just do something like termination.patch but >to >conditionalize crashing the pool on a pool configuration option. >That way >the behavior would no worse for your use case. Does that >sound reasonable? How would you shut down the pool then? And why is that simpler? -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: Btw, the current problem with termination3.patch seems to be that the MainProcess somehow appears in self._pool. I have no idea how it gets there. Maybe some unrelated issue that appears when forking that late in the tests. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Ask Solem added the comment: > A potential implementation is in termination.patch. Basically, > try to shut down gracefully, but if you timeout, just give up and > kill everything. You can't have a sensible default timeout, because the worker may be processing something important... > It's a lot less code (one could write an even shorter patch > that doesn't try to do any additional graceful error handling), > doesn't add a new monitor thread, doesn't add any more IPC > mechanism, etc.. FWIW, I don't see any of these changes as bad, > but I don't feel like I have a sense of how generally useful they > would be. Not everything can be simple. Getting this right may require a bit of code. I think we can get rid of the ack_handler thread by making the result handler responsible for both acks and results, but I haven't tried it yet, and this code is already running in production by many so didn't want to change it unless I had to. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
