[issue37754] alter size of segment using multiprocessing.shared_memory

Davin Potts added the comment: Attempts to alter the size of a shared memory segment are met with a variety of different, nuanced behaviors on systems we want to support. I agree that it would be valuable to be able to effectively realloc a shared memory segment, which thankfully the user

[issue37754] Consistency of Unix's shared_memory implementation with windows

Davin Potts added the comment: A shared semaphore approach for the resource tracker sounds appealing as a way to make the behavior on Windows and posix systems more consistent. However this might get implemented, we should not artificially prevent users from having some option to persist

[issue37185] use os.memfd_create in multiprocessing.shared_memory?

Davin Potts added the comment: Unless I am missing something, memfd_create appears to be specific to the Linux kernel still so we would need to replicate its behavior on all of the other unix systems. To your point, but quoting from the docs, "separate invocations of memfd_create wit

[issue38018] Increase Code Coverage for multiprocessing.shared_memory

Davin Potts added the comment: New changeset d14e39c8d9a9b525c7dcd83b2a260e2707fa85c1 by Davin Potts (Vinay Sharma) in branch 'master': bpo-38018: Increase code coverage for multiprocessing.shared_memory (GH-15662) https://github.com/python/cpyt

[issue38018] Increase Code Coverage for multiprocessing.shared_memory

Davin Potts added the comment: Initial review of the test failure suggests a likely flaw in the mechanism used by the resource tracker. I will continue investigating more tomorrow. -- ___ Python tracker <https://bugs.python.org/issue38

[issue38084] multiprocessing cannot recover from crashed worker

Davin Potts added the comment: Sharing for the sake of documenting a few things going on in this particular example: * When a PoolWorker process exits in this way (os._exit(anything)), the PoolWorker never gets the chance to send a signal of failure (normally sent via the outqueue) to the

[issue38084] multiprocessing cannot recover from crashed worker

Davin Potts added the comment: Thanks to Pablo's good work with implementing the use of multiprocessing's Process.sentinel, the logic for handling PoolWorkers that die has been centralized into Pool._maintain_pool(). If _maintain_pool() can also identify which job died wit

[issue38084] multiprocessing cannot recover from crashed worker

Davin Potts added the comment: Agreed with @ppperry that this is a duplicate of issue22393. The proposed patch in issue22393 is, for the moment, out of sync with more recent changes. That patch's approach would result in the loss of all partial results from a Pool.map, but it may be f

[issue35267] reproducible deadlock with multiprocessing.Pool

Davin Potts added the comment: I second what @vstinner already said in the comments for PR11143, that this should not merely be documented. -- nosy: +davin ___ Python tracker <https://bugs.python.org/issue35

[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access

New submission from Davin Potts : The resource tracker currently destroys (via _posixshmem.shm_unlink) shared memory segments on posix systems when any independently created Python process with a handle on a shared memory segment exits (gracefully or otherwise). This breaks the expected

[issue37754] Consistency of Unix's shared_memory implementation with windows

Davin Potts added the comment: I have created issue38119 to track a fix to the inappropriate use of resource tracker with shared memory segments, but this does not replace or supersede what is discussed here. -- ___ Python tracker <ht

[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access

Change by Davin Potts : -- keywords: +patch pull_requests: +15618 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15989 ___ Python tracker <https://bugs.python.org/issu

[issue37637] multiprocessing numpy.ndarray not transmitted properly

Davin Potts added the comment: Marking as closed after providing an example of how to send NumPy arrays as bytes with the send_bytes() function. -- resolution: -> not a bug stage: -> resolved status: -> closed ___ Python tracke

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Davin Potts added the comment: Short responses to questions/comments from @bjs, followed by hopefully helpful further comments: > Are you supposed to ever use a raw SharedMemory buffer directly? Yes. > What atomicity guarantees are there for ShareableList operations and > read

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Davin Potts added the comment: Apologies, one of the quotes in my previous response should have been attributed to @mental. I think @pierreglaser phrased it very nicely: > shared_memory is a low level python module. Precautions should be made when > handling concurrently the shared_

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Change by Davin Potts : -- pull_requests: +15722 pull_request: https://github.com/python/cpython/pull/16103 ___ Python tracker <https://bugs.python.org/issue22

[issue35727] sys.exit() in a multiprocessing.Process does not align with Python behavior

Davin Potts added the comment: I believe the mentality behind multiprocessing.Process triggering an exit code of 1 when sys.exit() is invoked inside its process is to indicate a non-standard exit out of its execution. There may yet be other side effects that could be triggered by having a

[issue33082] multiprocessing docs bury very important 'callback=' args

Davin Potts added the comment: I appreciate the functionality offered by the callbacks and have found good uses for them, as Chad clearly does/has. That said, the thought of expanding the documentation on the callbacks had not come up for me. Reading through the proposed changes to the

[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

Davin Potts added the comment: My sense is that it would be nice if we can catch this before ftruncate does something nasty. Where else is ftruncate used in CPython that this could similarly trigger a problem? How is it handled there (or not

[issue40440] allow array.array construction from memoryview w/o copy

Davin Potts added the comment: Being able to create an array.array without making a copy of a memoryview's contents does sound valuable. We do not always want to modify the size of the array, as evidenced by array.array's existing functionality where its size-changing manipulat

[issue20854] multiprocessing.managers.Server: problem with returning proxy of registered object

Davin Potts added the comment: It appears that the multiple workarounds proposed by the OP (@allista) address the original request and that there is no bug or unintended behavior arising from multiprocessing itself. Combined with the lack of activity in this discussion, I'm inclin

[issue31308] forkserver process isn't re-launched if it died

Davin Potts added the comment: I have two concerns with this: 1) The implicit restart of the forkserver process seems in conflict with the zen of making things explicit. 2) This would seem to make forkserver's behavior inconsistent with the behavior of things like the Manager which simi

[issue33725] Python crashes on macOS after fork with no exec

Davin Potts added the comment: Do we really need to disable the running of test_multiprocessing_fork entirely on MacOS? My understanding so far is that not *all* of the system libraries on the mac are spinning up threads and so we should expect that there are situations where fork alone

[issue33725] Python crashes on macOS after fork with no exec

Davin Potts added the comment: @ned.deily: Apologies, I misread what you wrote -- I would like to see the random segfaults that you were seeing on Mojave if you can still point me to a few. -- ___ Python tracker <https://bugs.python.

[issue35813] shared memory construct to avoid need for serialization between processes

New submission from Davin Potts : A facility for using shared memory would permit direct, zero-copy access to data across distinct processes (especially when created via multiprocessing) without the need for serialization, thus eliminating the primary performance bottleneck in the most

[issue35813] shared memory construct to avoid need for serialization between processes

Change by Davin Potts : -- keywords: +patch pull_requests: +11470 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Py

[issue35813] shared memory construct to avoid need for serialization between processes

Change by Davin Potts : -- keywords: +patch, patch, patch pull_requests: +11470, 11471, 11472 stage: -> patch review ___ Python tracker <https://bugs.python.org/issu

[issue35813] shared memory construct to avoid need for serialization between processes

Change by Davin Potts : -- keywords: +patch, patch pull_requests: +11470, 11471 stage: -> patch review ___ Python tracker <https://bugs.python.org/issu

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: New changeset e5ef45b8f519a9be9965590e1a0a587ff584c180 by Davin Potts in branch 'master': bpo-35813: Added shared_memory submodule of multiprocessing. (#11664) https://github.com/python/cpython/commit/e5ef45b8f519a9be9965590e1a0a58

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: This work is the result of ~1.5 years of development effort, much of it accomplished at the last two core dev sprints. The code behind it has been stable since September 2018 and tested as an independently installable package by multiple people. I was

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: @lukasz.langa: Missing tests and documentation will be in by alpha2. -- ___ Python tracker <https://bugs.python.org/issue35

[issue35903] Build of posixshmem.c should probe for required OS functions

Davin Potts added the comment: Agreed that the logic for building that code needs exactly this sort of change. Thanks for the patch! It looks like your patch does not happily detect the dependencies on MacOS for some reason, but all appears well on Windows & Linux. I will have a cl

[issue35813] shared memory construct to avoid need for serialization between processes

Change by Davin Potts : -- pull_requests: +11834, 11835 ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsub

[issue35813] shared memory construct to avoid need for serialization between processes

Change by Davin Potts : -- pull_requests: +11834, 11835, 11836 ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsub

[issue35813] shared memory construct to avoid need for serialization between processes

Change by Davin Potts : -- pull_requests: +11834 ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: Docs and tests are now available in a new PR. I have stayed focused on getting these docs and tests to everyone without delay but that means I have not yet had an opportunity to respond to the helpful comments, thoughtful questions, and threads that have

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: @giampaolo.rodola: Your patch from 3 days ago in issue35917 included additional tests around the SharedMemoryManager which are now causing test failures in my new PR. This is my fault because I altered SharedMemoryManager to no longer support functionality

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: @terry.reedy and @ronaldoussoren: I have asked Van again to provide comments here clarifying the topics of (1) copyright notices and (2) requiring the BSD-licensed-work's author to sign a contributor agreement. Specifically regarding the appearan

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: @giampaolo.rodola: It definitely helps. Conceptually, SyncManager provides "distributed shared memory" where lists, dicts, etc. are held in memory by one process but may be accessed remotely from another via a Proxy Object. Mutating a dict from o

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: @Antoine: SharedMemoryManager does not subclass SyncManager but it did previously. This is the source of the confusion. SharedMemoryManager subclasses BaseManager which does not provide Value, Array, list, dict, etc. Agreed that the manager facility does

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: These questions (originally asked in comments on GH-11816) seemed more appropriate to discuss here: Why should the user want to use `SharedMemory` directly? Why not just go through the manager? Also, perhaps a naive question: don't you _always_

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: Regarding the API of the SharedMemory class, its flags, mode, and read_only parameters are not universally enforced or simply not implemented on all platforms that offer POSIX Shared Memory or Windows Named Shared Memory. A simplified API for the SharedMemory

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: @giampaolo: > 1) it seems SharedMemory.close() does not destroy the memory region > (I'm able to re-attach to it via name). If I'm not mistaken only > the manager can do that. Correct, close() does not and should not destroy the memory re

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: @giampaolo: > Maybe something like this instead? > SharedMemory(name=None, attach_if_exists=False, size=0) I think this misses the use case when wanting to ensure we only attach to an existing shared memory block and if it does not exist, we should

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: @giampaolo: > 1) As for SharedMemoryManager, I believe it should live in > multiprocessing.managers, not shared_memory.py. I am happy to see it live in multiprocessing.managers so long as we can provide a clean way of handling what happens on a platform

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: @giampaolo: > Also, what happens if you alter the size of an existing object with a smaller > value? Is the memory region overwritten? Attaching to an existing shared memory block with a size=N which is smaller than its allocated size (say it was create

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: > I think we need the "create with exclusive behavior" option, even > though we don't know how to implement it on Windows right now. A fix to avoid the potential race condition on Windows is now part of GH-11816. > To support 1 & 2

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: The simpler API is now implemented in GH-11816 as discussed previously. Notably: > * We go with this simpler API: SharedMemory(name=None, create=False, size=0) > * 'size' is ignored when create=False > * create=True acts like O_CREX an

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: > Code looks much better now. I'm still not convinced > "SharedMemory(name=None, create=False, size=0)" is the best API. > How are you supposed to "create or attach" atomically? We are consciously choosing to not support a

[issue29454] Shutting down consumer on a remote queue

Davin Potts added the comment: My understanding is that example uses a queue.Queue() to demonstrate how to create a custom, remote service from scratch. The implementation in this simple example lacks the sophistication of multiprocessing.Queue() for handling situations such as the one

[issue29454] Shutting down consumer on a remote queue

Changes by Davin Potts : -- stage: -> needs patch type: behavior -> enhancement ___ Python tracker <http://bugs.python.org/issue29454> ___ ___ Python-bugs-

[issue29454] Shutting down consumer on a remote queue

Davin Potts added the comment: My understanding of other message queueing systems is that many are motivated by speed to the point that they will permit messages to be "lost" due to specific scenarios that would be overly costly to defend against. Other message queueing syste

[issue29701] Add close method to queue.Queue

Davin Potts added the comment: The example of AMQP is perhaps a stronger argument for why multiprocessing.Queue.close should (or does) exist, not as much a reason for queue.Queue. The strongest point, I think, is the argument that existing patterns are lacking. In the multiprocessing module

[issue29795] Clarify how to share multiprocessing primitives

Davin Potts added the comment: On Windows, because that OS does not support fork, multiprocessing uses spawn to create new processes by default. Note that in Python 3, multiprocessing provides the user with a choice of how to create new processes (i.e. fork, spawn, forkserver). When fork is

[issue17560] problem using multiprocessing with really big objects?

Davin Potts added the comment: @artxyz: The current release of 2.7 is 2.7.13 -- if you are still using 2.7.5 you might consider updating to the latest release. As pointed out in the text of the issue, the multiprocessing pickler has been made pluggable in 3.3 and it's been made

[issue29795] Clarify how to share multiprocessing primitives

Changes by Davin Potts : -- resolution: -> works for me stage: needs patch -> resolved status: open -> closed ___ Python tracker <http://bugs.python.or

[issue29828] Allow registering after-fork initializers in multiprocessing

Davin Potts added the comment: Having a read through issue16500 and issue6721, I worry that this could again become bogged down with similar concerns. With the specific example of NumPy, I am not sure I would want its random number generator to be reseeded with each forked process. There are

[issue30018] multiprocessing.Pool garbles call stack for __new__

Davin Potts added the comment: It looks like the first 'Called Foo.__new__' is being reported by the child (pool of 1) process and the second 'Called Foo.__new__' is being reported by the parent process. In multiprocessing, because objects are by default serialized using

[issue30018] multiprocessing.Pool garbles call stack for __new__

Davin Potts added the comment: Expanding my above example to show how multiprocessing relates: >>> import multiprocessing >>> import os >>> class Floof(object): ... def __new__(cls): ... print("New via pid=%d" % os.getpid()) ... return o

[issue30018] multiprocessing.Pool garbles call stack for __new__

Davin Potts added the comment: > I am unfortunately not at liberty to share the code I'm working on. I very much understand and am very thankful you took the time to create a simple example that you could share. Honestly, that's the reason I felt inspired to stop what I was doi

[issue30379] multiprocessing Array create for ctypes.c_char, TypeError unless 1 char string arg used

Davin Potts added the comment: Maybe I missed your point but why would you not want to do this instead? >>> mp.Array(ctypes.c_int8, arr) > -- nosy: +davin ___ Python tracker <http://bugs.python

[issue30379] multiprocessing Array create for ctypes.c_char, TypeError unless 1 char string arg used

Davin Potts added the comment: Perhaps I should've used ctypes.c_uint8 in that example/question instead. -- ___ Python tracker <http://bugs.python.org/is

[issue26434] multiprocessing cannot spawn grandchild from a Windows service

Davin Potts added the comment: New changeset c47c315812b1fa9acb16510a7aa3b37d113def48 by Davin Potts (Marc Schlaich) in branch '2.7': bpo-26434: Fix multiprocessing grandchilds in a Windows service (GH-1167) https://github.com/python/cpython/commit/c47c315812b1fa9acb16510a7aa3b3

[issue26434] multiprocessing cannot spawn grandchild from a Windows service

Davin Potts added the comment: Patch committed in 2.7 branch. Thanks for your help, Marc. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <http://bugs.python.or

[issue28053] parameterize what serialization is used in multiprocessing

Davin Potts added the comment: Docs need updating still. -- versions: +Python 3.7 ___ Python tracker <http://bugs.python.org/issue28053> ___ ___ Python-bugs-list m

[issue30317] test_timeout() of test_multiprocessing_spawn.WithManagerTestBarrier fails randomly on x86 Windows7 3.x buildbot

Changes by Davin Potts : -- pull_requests: +1810 ___ Python tracker <http://bugs.python.org/issue30317> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue30317] test_timeout() of test_multiprocessing_spawn.WithManagerTestBarrier fails randomly on x86 Windows7 3.x buildbot

Davin Potts added the comment: To better accommodate very slow buildbots, a parameter is added in PR-1722 to scale up the timeout durations where they are necessary on a per-machine basis. Relevant tests have a timeout set to some default number of seconds times a multiplier value. The

[issue30339] test_multiprocessing_main_handling: "RuntimeError: Timed out waiting for results" on x86 Windows7 3.x

Davin Potts added the comment: Patch on issue30317 also addresses this issue in a more flexible way. -- dependencies: +test_timeout() of test_multiprocessing_spawn.WithManagerTestBarrier fails randomly on x86 Windows7 3.x buildbot nosy: +davin

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: > FWIW I bumped into this lib: http://semanchuk.com/philip/sysv_ipc/ The author of that lib, Philip Semanchuk, is one of the people participating in this effort -- he has posted above in msg334934 here on b.p.o. and has helped review the PR in GH-11816. He

[issue36018] Add a Normal Distribution class to the statistics module

Davin Potts added the comment: There is an inconsistency worth paying attention to in the choice of names of the input parameters. Currently in the statistics module, pvariance() accepts a parameter named "mu" and pstdev() and variance() each accept a parameter named "x

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: New changeset e895de3e7f3cc2f7213b87621cfe9812ea4343f0 by Davin Potts in branch 'master': bpo-35813: Tests and docs for shared_memory (#11816) https://github.com/python/cpython/commit/e895de3e7f3cc2f7213b87621cfe98

[issue35813] shared memory construct to avoid need for serialization between processes

Davin Potts added the comment: @Giampaolo: The docstring in the shared_memory module currently marks the API as experimental. (You read my mind...) I will start a new PR where we can work on the better-integration-into-the-larger-multiprocessing-docs and add comments there

[issue36018] Add a Normal Distribution class to the statistics module

Davin Potts added the comment: Steven: Your point about population versus sample makes sense and your point that altering their names would be a breaking change is especially important. I think that pretty well puts an end to my suggestion of alternative names and says the current pattern

[issue36099] Clarify the difference between mu and xbar in the statistics documentation

Davin Potts added the comment: Without necessarily defining what each means, perhaps it is sufficient to change this clause in the docs: it should be the mean of data For pvariance() it could read as: it should be the *population* mean of data And for variance() it could read as

[issue36102] TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x

Davin Potts added the comment: Though apparently undocumented on FreeBSD, their implementation of shm_open differs from others in the following way: all names for shared memory blocks *must* begin with a slash. This requirement does not exist on OpenBSD. According to its man page on

[issue36102] TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x

Davin Potts added the comment: In local testing, I found the following systems to impose the leading slash as a requirement for simply creating a shared memory block: * NetBSD 8.0 * FreeBSD 12.x * TrueOS 18.12 (the OS formerly known as PC-BSD) I found the following systems to have no

[issue36102] TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x

Change by Davin Potts : -- keywords: +patch pull_requests: +12065 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36102> ___ ___ Py

[issue36102] TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x

Davin Potts added the comment: I have locally tested GH-12036 on all 5 of the aforementioned OSes and all are made happy by the patch. Victor: If we want to go ahead and apply this patch right away to hopefully make the FreeBSD buildbot go green, the nature of this change is sufficiently

[issue36102] TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x

Change by Davin Potts : -- stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36102> ___ ___ Python-bugs-list mailing list Unsubscrib

[issue36102] TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x

Davin Potts added the comment: New changeset aadef2b41600cb6a4f845cdc4cea001c916d8745 by Davin Potts in branch 'master': bpo-36102: Prepend slash to all POSIX shared memory block names (#12036) https://github.com/python/cpython/commit/aadef2b41600cb6a4f845cdc4cea00

[issue36102] TestSharedMemory fails on AMD64 FreeBSD CURRENT Shared 3.x

Davin Potts added the comment: Closing. Thank you Giampaolo for jumping in so quickly to review! Thank you Victor for catching this on the buildbot. Though what is this talk of "_if_ the color changes"? ;) -- resolution: -> fixed stage: patch review -> resol

[issue33725] Python crashes on macOS after fork with no exec

Davin Potts added the comment: As best as I can see, there is no magic bullet to help mitigate this. At a minimum, I am convinced we need to update the documentation to describe this behavior on MacOS and recommend alternatives. I continue to give serious thought to the idea of changing the

[issue36364] errors in multiprocessing.shared_memory examples

Davin Potts added the comment: Very much agreed, they're moving over to the main docs. -- ___ Python tracker <https://bugs.python.org/issue36364> ___ ___

[issue33725] Python crashes on macOS after fork with no exec

Davin Potts added the comment: I believe we must change the default behavior on MacOS to use spawn instead of fork. Encouraging people to use fork by default on MacOS is encouraging them to create something that effectively will not work. Keeping fork as the default behavior when we have

[issue33725] Python crashes on macOS after fork with no exec

Davin Potts added the comment: Victor raises an important question: should the *default* start behavior be made consistent across platforms? Assuming we change it on MacOS, the default start behavior on Windows and MacOS will be spawn but the default start behavior on Linux and FreeBSD

[issue33196] multiprocessing: serialization must ensure that contexts are compatible (the same)

Change by Davin Potts : -- nosy: +davin ___ Python tracker <https://bugs.python.org/issue33196> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished

Change by Davin Potts : -- nosy: +davin ___ Python tracker <https://bugs.python.org/issue35242> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33725] Python crashes on macOS after fork with no exec

Davin Potts added the comment: Barry's effort as well as comments in other links seem to all suggest that OBJC_DISABLE_INITIALIZE_FORK_SAFETY is not comprehensive in its ability to make other threads "safe" before forking. "Objective-C classes defined by the OS framework

[issue33725] Python crashes on macOS after fork with no exec

Davin Potts added the comment: Given the original post mentioned 2.7.15, I wonder if it is feasible to fork near the beginning of execution, then maintain and pass around a multiprocessing.Pool to be used when needed instead of dynamically forking? Working with legacy code is almost always

[issue35219] macOS 10.14 Mojave crashes in multiprocessing

Davin Potts added the comment: Resolution is marked dupe but status is still open. Are we closing this one or is there a more specific remedy for this situation (as opposed to what issue33725 discusses) that would be helpful to document? -- nosy: +davin

[issue28699] Imap from ThreadPool behaves unexpectedly

Davin Potts added the comment: @xiang.zhang: Nice catch -- thank you for pointing out the additional issue that arises when trying to provoke the issue twice in a row. The module attribute `job_counter` helped, in this situation, point out what might have otherwise been overlooked. I didn&#

[issue28699] Imap from ThreadPool behaves unexpectedly

Davin Potts added the comment: Attaching promised crude tests. -- Added file: http://bugs.python.org/file45646/issue_28699_repro.py ___ Python tracker <http://bugs.python.org/issue28

[issue28625] multiprocessing.Pool.imap swallows exceptions thrown by generators

Davin Potts added the comment: Closing this issue -- all further discussion moves to issue28699 -- resolution: -> duplicate stage: -> resolved status: open -> closed ___ Python tracker <http://bugs.python.or

[issue28696] imap from ThreadPool hangs by an exception in a generator function

Davin Potts added the comment: Closing this issue -- all further discussion moves to issue28699 -- resolution: -> duplicate stage: -> resolved status: open -> closed ___ Python tracker <http://bugs.python.or

[issue28779] set_forkserver_preload() can crash the forkserver if preloaded module instantiate multiprocessing classes

Davin Potts added the comment: Antoine: I'm still unclear on what's going on here but I noticed something by accident when trying your example script. Specifically, when I ran your script from the directory where it exists on disk, I could successfully reproduce what you describe

[issue28779] set_forkserver_preload() can crash the forkserver if preloaded module instantiate multiprocessing classes

Davin Potts added the comment: I don't see any negative consequences for the helpers if the `force=True` is made in spawn.prepare's invocation of set_start_method(). In tracing backwards to figure out why this wasn't done already, it seems unchanged since sbt's patch i

[issue28973] The fact that multiprocess.Queue uses serialization should be documented.

Davin Potts added the comment: All communication between processes in multiprocessing has consistently used pickle to serialize the data being communicated (this includes what is described in the "Shared memory" section of the docs). The documentation has not done a great job of m

[issue28965] Multiprocessing spawn/forkserver fails to pass Queues

Changes by Davin Potts : -- nosy: +davin type: crash -> behavior ___ Python tracker <http://bugs.python.org/issue28965> ___ ___ Python-bugs-list mailing list Un

[issue29108] Python 3.6.0 multiprocessing map_async callback

Davin Potts added the comment: This appears to be working as designed. The purpose of the callback is to receive a single argument (nominally, the ready result from the map operation) and perform its task (whatever it may be) very quickly so as not to further delay the handing back of

[issue29158] Possible glitch in the interaction of a thread and a multiprocessing manager

Davin Potts added the comment: There are too many things going on in this example -- it would be far easier to digest if the example could be simplified. The general programming rule of thumb (completely unrelated to but still just as relevant to Python) that I think David might have been

  1   2   3   4   >