[Python-Dev] contributing to multiprocessing
Hi all -- I am interested in making some serious ongoing contributions around multiprocessing. My inspiration, first and foremost, comes from the current documentation for multiprocessing. There is great material there but I believe it is being presented in a way that hinders adoption and understanding. I've taken some initial baby-steps to propose specific changes: http://bugs.python.org/issue22952 http://bugs.python.org/issue23100 The first, issue22952, can reasonably be tackled with a patch like I've submitted. Continuing with patches for issue23100 can also be made to work. I realize that reviewing such patches takes non-trivial time from volunteers yet I'm interested in submitting a series of patches to hopefully make the documentation for multiprocessing much more consistent with other module docs and much more accessible to end users. I don't want to simply create more work for other volunteers -- I'd like to volunteer to reduce / share some of their work as well. Beyond the documentation, there is currently a backlog of 186 issues mentioning multiprocessing, some with patches on offer, some without. I'd like to volunteer my time reviewing and triaging these issues. Hopefully you can already get a sense of my voice on issues from what I wrote in those two issues above. Rather than me simply walking through that backlog, offering comments or encouragement here and there on issues, it makes more sense for me to ask: what is the right way for me to proceed? What is the next step towards me helping triage issues? Is there a bridge-keeper with at least three, no more than five questions for me? Thanks, Davin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] contributing to multiprocessing
Thanks! That sounds like a nice, clear path forward. Regarding the doc issues being a bit more problematic to work through, I thoroughly understand. Davin On Jan 8, 2015, at 21:19, R. David Murray wrote: > On Thu, 08 Jan 2015 17:08:07 -0800, Ethan Furman wrote: >> On 01/08/2015 03:21 PM, Davin Potts wrote: >>> >>> I am interested in making some serious ongoing contributions around >>> multiprocessing. >> >> Great! >> >>> Rather than me simply walking through that backlog, offering comments or >>> encouragement here and there on issues, it >>> makes more sense for me to ask: what is the right way for me to proceed? >>> What is the next step towards me helping >>> triage issues? Is there a bridge-keeper with at least three, no more than >>> five questions for me? >> >> I would suggest having at least one, if not two or three, current core-devs >> ready and willing to quickly review your >> work (I believe Raymond Hettinger may be one); then, go ahead and triage, >> improve and/or submit patches, and make >> comments. Once you've got a few of these under your belt, with favorable >> reviews and your patches committed, you may >> get stuck with commit privileges of your own. ;) > > Indeed, the best way to proceed, regardless of any other issues, is in > fact to review, triage, comment on, and improve the issues you are > interested in. Get the patches commit ready, and then get a current > core dev to do a commit review. > > Oddly, the doc issues may be more problematic than the code issues. > Fixing bugs in docs isn't difficult to get done, but restructuring > documentation sometimes gets bogged down in differing opinions. (I > haven't myself looked at your proposals since I don't use > multiprocessing, so I don't know how radical the proposed changes are). > > --David > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/python%2Bpython_dev%40discontinuity.net ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OpenBSD buildbot has many failures
Hi Victor — I am personally interested in seeing all tests pass on OpenBSD and am willing to put forth effort to help that be so. I would be happy to be added to any issues that get opened against OpenBSD. That said, I have concerns about the nature of when and how these failures came about — specifically I worry that other devs have committed the changes which prompted these failures yet they did not pay attention nor take responsibility when it happened. Having monitored certain buildbots for a while to see how the community behaves and devs fail to react when a failure is triggered by a commit, I think we should do much better in taking individual responsibility for prompting these failures. Davin > On Mar 28, 2015, at 04:53, Victor Stinner wrote: > > Hi, > > The OpenBSD buildbot always fail with the same failures since many > months (ex: test_socket). Is someone interested to fix them? If no, > would it be possible to turn it off to get a better view of > regressions? Currently, they are too many red buildbots to quickly see > regressions introduced by recent commits. > > http://buildbot.python.org/all/builders/x86%20OpenBSD%205.5%203.x > > Victor > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/python%2Bpython_dev%40discontinuity.net ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Asking for reversion
I am attempting to do the right thing and am following the advice of other core devs in what I have done thus far. Borrowing heavily from what I've added to issue35813 just now: This work is the result of ~1.5 years of development effort, much of it accomplished at the last two core dev sprints. The code behind it has been stable since September 2018 and tested as an independently installable package by multiple people. I was encouraged by Lukasz, Yury, and others to check in this code early, not waiting for tests and docs, in order to both solicit more feedback and provide for broader testing. I understand that doing such a thing is not at all a novelty. Thankfully it is doing that -- I hope that feedback remains constructive and supportive. There are some tests to be found in a branch (enh-tests-shmem) of github.com/applio/cpython which I think should become more comprehensive before inclusion. Temporarily deferring and not including them as part of the first alpha should reduce the complexity of that release. Regarding the BSD license on the C code being adopted, my conversations with Brett and subsequently Van have not raised concerns, far from it -- there is a process which is being followed to the letter. If there are other reasons to object to the thoughtful adoption of code licensed like this one, that deserves a decoupled and larger discussion first. Davin On Sun, Feb 3, 2019 at 8:12 PM Raymond Hettinger < raymond.hettin...@gmail.com> wrote: > > > On Feb 3, 2019, at 5:40 PM, Terry Reedy wrote: > > > > On 2/3/2019 7:55 PM, Guido van Rossum wrote: > >> Also, did anyone ask Davin directly to roll it back? > > > > Antoine posted on the issue, along with Robert O. Robert reviewed and > make several suggestions. > > I think the PR sat in a stable state for many months, and it looks like > RO's review comments came *after* the commit. > > FWIW, with dataclasses we decided to get the PR committed early, long > before most of the tests and all of the docs. The principle was that bigger > changes needed to go in as early as possible in the release cycle so that > we could thoroughly exercise it (something that almost never happens while > something is in the PR stage). It would be great if the same came happen > here. IIRC, shared memory has long been the holy grail for > multiprocessing, helping to mitigate its principle disadvantage (the cost > of moving data between processes). It's something we really want. > > But let's see what the 3.8 release manager has to say. > > > Raymond > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/python%2Bpython_dev%40discontinuity.net > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Asking for reversion
On 2/3/2019 7:55 PM, Guido van Rossum wrote: > Also, did anyone ask Davin directly to roll it back? Simply put: no. There have been a number of reactionary comments in the last 16 hours but no attempt to reach out to me directly during that time. On Sun, Feb 3, 2019 at 8:12 PM Raymond Hettinger < raymond.hettin...@gmail.com> wrote: > It was only recently that you edited his name out of the list of maintainers for multiprocessing > even though that is what he's been working on for the last two years and at the last two sprints. I think it would be best to discuss Antoine's decision to take this particular action without first consulting me, elsewhere and not part of this thread. As I said, I am happy to do the most constructive thing possible and I sought the advice of those I highly respect first before doing as I have. Davin On Sun, Feb 3, 2019 at 9:12 PM Davin Potts < python+python_...@discontinuity.net> wrote: > I am attempting to do the right thing and am following the advice of other > core devs in what I have done thus far. > > Borrowing heavily from what I've added to issue35813 just now: > > This work is the result of ~1.5 years of development effort, much of it > accomplished at the last two core dev sprints. The code behind it has been > stable since September 2018 and tested as an independently installable > package by multiple people. > > I was encouraged by Lukasz, Yury, and others to check in this code early, > not waiting for tests and docs, in order to both solicit more feedback and > provide for broader testing. I understand that doing such a thing is not > at all a novelty. Thankfully it is doing that -- I hope that feedback > remains constructive and supportive. > > There are some tests to be found in a branch (enh-tests-shmem) of > github.com/applio/cpython which I think should become more comprehensive > before inclusion. Temporarily deferring and not including them as part of > the first alpha should reduce the complexity of that release. > > Regarding the BSD license on the C code being adopted, my conversations > with Brett and subsequently Van have not raised concerns, far from it -- > there is a process which is being followed to the letter. If there are > other reasons to object to the thoughtful adoption of code licensed like > this one, that deserves a decoupled and larger discussion first. > > > Davin > > On Sun, Feb 3, 2019 at 8:12 PM Raymond Hettinger < > raymond.hettin...@gmail.com> wrote: > >> >> > On Feb 3, 2019, at 5:40 PM, Terry Reedy wrote: >> > >> > On 2/3/2019 7:55 PM, Guido van Rossum wrote: >> >> Also, did anyone ask Davin directly to roll it back? >> > >> > Antoine posted on the issue, along with Robert O. Robert reviewed and >> make several suggestions. >> >> I think the PR sat in a stable state for many months, and it looks like >> RO's review comments came *after* the commit. >> >> FWIW, with dataclasses we decided to get the PR committed early, long >> before most of the tests and all of the docs. The principle was that bigger >> changes needed to go in as early as possible in the release cycle so that >> we could thoroughly exercise it (something that almost never happens while >> something is in the PR stage). It would be great if the same came happen >> here. IIRC, shared memory has long been the holy grail for >> multiprocessing, helping to mitigate its principle disadvantage (the cost >> of moving data between processes). It's something we really want. >> >> But let's see what the 3.8 release manager has to say. >> >> >> Raymond >> >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/python%2Bpython_dev%40discontinuity.net >> > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] About multiprocessing maintainership
Antoine's change to the devguide was made on the basis that "he doesn't contribute anymore" which, going by Antoine's own description in this thread, he contradicts. My current effort, mentioned in Antoine's other thread, is not my single largest contribution. I have been impressed by the volume of time that Antoine is able to spend on the issue tracker. Because he and I generally agree on what actions to take on an issue, when he is quick to jump on issues it is uncommon for me to feel the need to say something myself just to play a numbers game or to make my presence felt -- that's part of being a team. There have been incidents where I disagree with Antoine and explain the reasoning in an issue but later Antoine goes on to do whatever he wants, disregarding what I wrote -- because I am not in a position to necessarily react or respond as frequently, I've generally only discovered this much later. I regard this latter interaction as unhealthy. I have been part of several group discussions (among core developers) now regarding how to balance the efforts of contributors with copious time to devote versus those which must be extra judicious in how they spend their more limited time. We recognize this as an ongoing concern and here it is again. If we are supportive of one another, we can find a way to work through such things. I joined the core developer team to help others and give back especially when it involved the then-neglected multiprocessing module. When I am personally attacked in a discussion on an issue by someone I do not know, it hurts and demoralizes me -- I know that all of the core developers experience this. When I am spending time on multiprocessing only to be surprised by a claim that I don't contribute anymore, it hurts and demoralizes me. When I read hand-picked statistics to support a slanted narrative designed to belittle my contributions, it hurts and demoralizes me. I know different people react to such things differently but in my case I have occasionally needed to take time away from cpython to detox -- in 2018, such incidents led to my losing more than a month of time more than once. Regarding support for one another: At the core developer sprint last year, I volunteered to remotely host Antoine on my laptop so that he could video-conference into the governance discussions we were having there. A few weeks later, Antoine is "editing me out" of the maintainers list without any further communication. If we only let the loudest people contribute then we lose the quiet contributors and push them out. Davin On Mon, Feb 4, 2019 at 4:39 AM Antoine Pitrou wrote: > > Hello, > > In a recent message, Raymond dramatically pretends that I would have > "edited out" Davin of the maintainers list for the multiprocessing > module. > > What I did (*) is different: I asked to mark Davin inactive and to stop > auto-assigning him on bug tracker issues. Davin was /still/ listed in > the experts list, along with me and others. IOW, there was no "editing > out". > > (*) https://github.com/python/devguide/pull/435 > > The reason I did this is simple: Davin does not do, and has almost > never done, any actual maintenance work on multiprocessing (if you are > not convinced, just go through the git history, and the PRs that were > merged in the ~4 last years). He usually does not respond to tracker > issues opened by users. He does not review PRs. The only sizable > piece of work he committed is, as I mentioned in the previous thread, > still untested and undocumented. > > Auto-assigning someone who never (AFAICT) responds to issues ultimately > does a disservice to users, whose complaints go unanswered; while other > people, who /do/ respond to users, are not aware of those stale issues. > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/python%2Bpython_dev%40discontinuity.net > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Asking for reversion
I have done what I was asked to do: I added tests and docs in a new
PR (GH-11816) as of Feb 10.
Since that time, the API has matured thanks to thoughtful feedback
from a number of active reviewers. At present, we appear to have
stabilized around an API and code that deserves to be exercised
further. To get that exercise and because there are currently no
outstanding objections, I am merging the PR to get it into the second
alpha. There will undoubtedly be further revisions and adjustments.
During this effort, I have received a surprising number of personal
emails expressing support and encouragement from people, most of whom
I have never met. Your kindness has been wonderful.
Davin
On Wed, Feb 6, 2019 at 10:58 AM Giampaolo Rodola' wrote:
>
>
>
> On Wed, Feb 6, 2019 at 12:51 PM Giampaolo Rodola' wrote:
>>
>>
>> Unless they are already there (I don't know) it would be good to have a full
>> set of unit-tests for all the register()ed types and test them against
>> SyncManager and SharedMemoryManager. That would give an idea on the real
>> interchangeability of these 2 classes and would also help writing a
>> comprehensive doc.
>
>
> In order to speed up the alpha2 inclusion process I created a PR which
> implements what said above:
> https://github.com/python/cpython/pull/11772
> https://bugs.python.org/issue35917
> Apparently SharedMemoryManager works out of the box and presents no
> differences with SyncManager, but the list type is not using ShareableList.
> When I attempted to register it with "SharedMemoryManager.register('list',
> list, ShareableList)" I got the following error:
>
> Traceback (most recent call last):
> File "foo.py", line 137, in test_list
> o = self.manager.list()
> File "/home/giampaolo/svn/cpython/Lib/multiprocessing/managers.py", line
> 702, in temp
> proxy = proxytype(
> TypeError: __init__() got an unexpected keyword argument 'manager'
>
> I am not sure how to fix that (I'll leave it up to Davin). The tests as-is
> are independent from PR-11772 so I suppose they can be reviewed/checked-in
> regardless of the changes which will affect shared_memory.py.
>
> --
> Giampaolo - http://grodola.blogspot.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
