Brian Quinlan added the comment:
You'll probably get more traction if you file a new bug.
--
___
Python tracker
<http://bugs.python.org/issue5788>
___
___
Changes by Brian Quinlan :
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue11864>
___
___
Python-bugs-list
Brian Quinlan added the comment:
On my crappy computer, ProcessPoolExecutor.map adds <3ms of added execution
time per call using your test framework.
What is your use case where that is too much?
That being said, I don't have any objections to making improvements.
If you want to pur
Changes by Brian Quinlan :
--
assignee: -> bquinlan
___
Python tracker
<http://bugs.python.org/issue10028>
___
___
Python-bugs-list mailing list
Unsubscri
Brian Quinlan added the comment:
Hey Brian,
Could you try applying the patch that I attached and let me know what error
message you get?
Cheers,
(the other) Brian
--
keywords: +patch
Added file: http://bugs.python.org/file19137/error.diff
Brian Quinlan added the comment:
Fixed in r85288
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue9903>
___
__
Brian Quinlan added the comment:
I added myself as the maintainer of concurrent.futures.
I'll look at the patch now.
--
assignee: -> bquinlan
___
Python tracker
<http://bugs.python.org
Brian Quinlan added the comment:
Patch committed in r85567.
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/i
Brian Quinlan added the comment:
Looks good but you forgot to actually use your new Waiter ;-)
Committed as r86491.
--
resolution: -> fixed
stage: patch review -> committed/rejected
status: open -> closed
___
Python track
Brian Quinlan added the comment:
I would suggest that you base your patch on 3.3/default.
--
___
Python tracker
<http://bugs.python.org/issue11635>
___
___
Pytho
Brian Quinlan added the comment:
Your approach seems workable but your patch allows the interpreter to exit
while work items are still being processed. See the comment at the top of
concurrent/futures/thread.py.
--
___
Python tracker
<h
Brian Quinlan added the comment:
Sorry, I didn't read an error message very carefully. When I apply your patch I
see:
>>> from concurrent.futures import *
>>> from time import *
>>> t = ThreadPoolExecutor(5)
>>> t.submit(sleep, 100)
>>>
Erro
Brian Quinlan added the comment:
No, I wasn't able to replicate.
--
___
Python tracker
<http://bugs.python.org/issue10632>
___
___
Python-bugs-list m
Brian Quinlan added the comment:
This is the expected behavior.
Future.cancel() returns a bool indicating if the future was cancelled or not.
If it returns False then there is no way to stop it.
The behavior that you are seeing for shutdown is documented as:
"""
shutdown(wai
Brian Quinlan added the comment:
Filing a new bug might have been a mistake. Once the investigation in issue
10517 isolated the failure as being in a different module, I thought it best to
file a new bug with a minimal repro case.
Fill free to cleanup
New submission from Brian Quinlan :
from concurrent import futures
with futures.ThreadPoolExecutor(max_workers=5) as e:
e.map(print, range(10))
# No output
--
assignee: bquinlan
components: Library (Lib)
messages: 133104
nosy: bquinlan
priority: normal
severity: normal
status: open
Brian Quinlan added the comment:
I think that it surprising behavior, especially considering that asking for the
*first* element in the iterator causes *all* of the futures to be created.
--
___
Python tracker
<http://bugs.python.org/issue11
Brian Quinlan added the comment:
Nice catch. I hadn't noticed that the docs are lying :-)
--
___
Python tracker
<http://bugs.python.org/issue11777>
___
___
Changes by Brian Quinlan :
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue11777>
___
___
Python-bugs-list
Brian Quinlan added the comment:
It looks like the timing is too sensitive.
--
assignee: -> bquinlan
___
Python tracker
<http://bugs.python.org/issu
Brian Quinlan added the comment:
Under what circumstances do we expect a ProcessPoolExecutor child process to be
killed outside of the control of the ProcessPoolExecutor?
If the user kills a child then maybe all we want to do is raise an exception
rather than deadlock as a convenience
Brian Quinlan added the comment:
> Killed by the user, or by an automatic device (such as the Linux OOM
> killer), or crashed.
Crashed would be bad - it would indicate a bug in the
ProcessPoolExecutor code.
>
>> If the user kills a child then maybe all we want to do is raise
Brian Quinlan added the comment:
Would you include the entire output of the test run?
--
___
Python tracker
<http://bugs.python.org/issue10183>
___
___
Python-bug
Brian Quinlan added the comment:
Could you try rerunning the tests with the attached patch? It could be that
your machines are just terribly slow at starting new Python instances.
--
keywords: +patch
Added file: http://bugs.python.org/file19636/timing.patch
Brian Quinlan added the comment:
Could you try with the patch that I just attached? And thanks for you help, I
really appreciated it!
--
Added file: http://bugs.python.org/file19645/timing2.patch
___
Python tracker
<http://bugs.python.
Brian Quinlan added the comment:
Daniel,
I wasn't trying to avoid importing multiprocessing.
What's your use case though? I think that defaulting the number of threads to
the numbers of CPUs would trick users into believing that threads are useful
for CPU-intensive work
Brian Quinlan added the comment:
Fixed in r10183
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue10183>
___
__
Brian Quinlan added the comment:
I think that using the number of CPUs for max_workers makes sense for you but
won't for most users. So I wouldn't make it a default.
--
___
Python tracker
<http://bugs.python.o
Brian Quinlan added the comment:
I've attached a patch that removes the code that installs a handler to the
futures logger.
I'm not sure if this is the correct approach though - it means that
"impossible" errors will only be reported to the user through a message like
&
New submission from Brian Quinlan :
multiprocessing generates fatal error "Invalid thread state for this thread" in
PyThreadState_Swap
This seems to happen on RHEL 5 and Centos 5.5
Here is the minimal repro:
>>> import multiprocessing.managers
>>> mpp = mu
Brian Quinlan added the comment:
I've filed a new bug (http://bugs.python.org/issue10632) against
multiprocessing and this bug dependent on it.
In the meantime, I can't repro this on ubuntu 10.04 LTS so I'm going to install
Centos and give that a go.
Changes by Brian Quinlan :
--
title: multiprocessing gene -> multiprocessing generates a fatal error
___
Python tracker
<http://bugs.python.org/issu
Brian Quinlan added the comment:
Sorry for being AWOL for so long. Attached is a patch that doesn't install a
handler and checks stderr for the exception output. Unfortunately, it looks
like the logging tests are still messing things up:
./python.exe -m test test_concurrent_fu
Brian Quinlan added the comment:
What's the best way for me to test this? The problem occurs on a Windows-only
code path but there is not enough information for me to debug it.
Should I check-in some additional diagnostics, wait for the buildbot to run,
collect my data and then rollba
Brian Quinlan added the comment:
I'm good, thanks Brian C.
It looks like SetEvent is failing with ERROR_INVALID_HANDLE.
CRITICAL:root:SetEvent(2044) failed with 0, GetLastError() = 6
CRITICAL:root:SetEvent(2064) failed with 0, GetLastError() = 6
CRITICAL:root:SetEvent(2220) failed w
Brian Quinlan added the comment:
Fixed in r87556.
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue10626>
___
__
Brian Quinlan added the comment:
Martin,
Could you commit this patch if you think that it is the right thing? I'm going
to be restructuring the tests and don't want you to get caught in merge hell.
Cheers,
Brian
--
___
Python trac
Brian Quinlan added the comment:
The tests are currently flaky on Windows and I'd like to fix that before the
release. But I don't have the bandwidth to debug the Call() abstraction used in
the tests before 3.2 RC1.
You can see the test change here:
http://svn.python.org/view/pytho
Brian Quinlan added the comment:
Fixed in r87673.
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue10788>
___
__
Brian Quinlan added the comment:
Fixed in r87673.
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue10737>
___
__
Brian Quinlan added the comment:
Fixed in r87673.
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue10028>
___
__
Changes by Brian Quinlan :
--
stage: -> needs patch
___
Python tracker
<http://bugs.python.org/issue10632>
___
___
Python-bugs-list mailing list
Unsubscri
Brian Quinlan added the comment:
Arian,
This seems like such an unimportant edge case that I don't want to mess with
the API just to accommodate it.
If you really need to pass an "fn" keyword argument, use:
.submit(foo, 1, 2, **{'fn': bar})
--
resolution:
Brian Quinlan added the comment:
Good point! I'd suggest functools.partial.
--
___
Python tracker
<http://bugs.python.org/issue10918>
___
___
Python-bugs-l
Brian Quinlan added the comment:
Sorry for taking so long to reply - I was on holidays until today.
This is an incompatible API change (since people may be providing "fn" by
keyword) so we should probably hold off until 3.3.
I also don't really like that the signature for sub
Brian Quinlan added the comment:
ProcessPoolExecutor is not expected to work with any interactive shell on
Windows because it uses the multiprocessing module behind the scenes, which has
that limitation.
I'm reclassifying this as a documentation bug since either a reference to
New submission from Brian Quinlan :
```
from concurrent.futures import ProcessPoolExecutor
import time
t = ProcessPoolExecutor(max_workers=3)
t.map(time.sleep, [1,2,3])
t.shutdown(wait=False)
```
Results in this exception and then a hang (i.e. Python doesn't terminate):
```
Excepti
Change by Brian Quinlan :
--
keywords: +patch
pull_requests: +17601
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/18221
___
Python tracker
<https://bugs.python.org/issu
Brian Quinlan added the comment:
New changeset 884eb89d4a5cc8e023deaa65001dfa74a436694c by Brian Quinlan in
branch 'master':
bpo-39205: Tests that highlight a hang on ProcessPoolExecutor shutdown (#18221)
https://github.com/python/cpython/commit/884eb89d4a5cc8e023deaa65001dfa
Brian Quinlan added the comment:
I'll try to take a look at this before the end of the week, but I'm currently
swamped with other life stuff :-(
--
___
Python tracker
<https://bugs.python.o
Brian Quinlan added the comment:
Thanks for the patch!
1. The fetching the state feature seems reasonable but I think that explaining
the difference between CANCELLED and CANCELLED_AND_NOTIFIED is going to be
hard. Maybe you could look at how Doc/library/concurrent.futures.rst would need
to
Brian Quinlan added the comment:
Could you run just the test_concurrent_futures test, hit ctrl-C at the point
where it hangs, and send the traceback here?
--
___
Python tracker
<http://bugs.python.org/issue14
Changes by Brian Quinlan :
--
assignee: -> bquinlan
nosy: +bquinlan
___
Python tracker
<http://bugs.python.org/issue14119>
___
___
Python-bugs-list mai
Brian Quinlan added the comment:
I'm closing this since tbrink didn't respond to pitrou's comments.
--
resolution: -> out of date
___
Python tracker
<http://bugs.
Changes by Brian Quinlan :
--
nosy: +bquinlan
___
Python tracker
<http://bugs.python.org/issue14148>
___
___
Python-bugs-list mailing list
Unsubscribe:
Brian Quinlan added the comment:
I guess the question is: why do you need to know the state in that form?
--
___
Python tracker
<http://bugs.python.org/issue13
Brian Quinlan added the comment:
Joshua, I'm closing this since I haven't heard from you in a month. Please
re-open if you use case isn't handled by `initializer` and `initargs`.
--
assignee: -> bquinlan
resolution: -> out of date
stage: -> resolve
Brian Quinlan added the comment:
That's a super interesting bug! It looks like this issue is that your exception
can't be round-tripped using pickle i.e.
>>> class PoolBreaker(Exception):
... def __init__(self, num):
... super().__init__()
...
New submission from Brian Quinlan :
$ ./python.exe nopickle.py
TypeError: __init__() missing 1 required positional argument: 'num'
The issue is that the arguments passed to Exception.__init__ (via `super()`)
are collected into `args` and then serialized by pickle e.g.
>>&
Change by Brian Quinlan :
--
title: picke cannot dump exceptions subclasses with different super() args ->
picke cannot dump Exception subclasses with different super() args
___
Python tracker
<https://bugs.python.org/issu
Change by Brian Quinlan :
--
resolution: -> duplicate
superseder: -> picke cannot dump Exception subclasses with different super()
args
___
Python tracker
<https://bugs.python.org/i
Change by Brian Quinlan :
--
stage: -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue37208>
___
___
Python-bugs-list
Change by Brian Quinlan :
--
resolution: -> duplicate
stage: -> resolved
status: open -> closed
superseder: -> function changed when pickle bound method object
___
Python tracker
<https://bugs.python
Brian Quinlan added the comment:
New changeset 242c26f53edb965e9808dd918089e664c0223407 by Brian Quinlan in
branch 'master':
bpo-31783: Fix a race condition creating workers during shutdown (#13171)
https://github.com/python/cpython/commit/242c26f53edb965e9808dd918089e6
Change by Brian Quinlan :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Brian Quinlan added the comment:
I don't know what the backport policy is. The bug is only theoretical AFAIK
i.e. someone noticed it through code observation but it has not appeared in
the wild.
On Mon, Jul 1, 2019 at 3:25 PM Ned Deily wrote:
>
> Ned Deily added the comment
Brian Quinlan added the comment:
Can I add "needs backport to 3.8" and "needs backport to 3.7" labels now or
do I have to use cherry_picker at this point?
On Mon, Jul 1, 2019 at 3:55 PM Ned Deily wrote:
>
> Ned Deily added the comment:
>
> > I don't
Brian Quinlan added the comment:
>> The current behavior is explicitly documented, so presumably
>> it can't be (easily) changed
And it isn't clear that it would be desirable to change this even if it were
possible - doing structured resource clean-up seems consi
Brian Quinlan added the comment:
If no one has short-term plans to improve multiprocessing.connection.wait, then
I'll update the docs to list this limitation, ensure that ProcessPoolExecutor
never defaults to >60 processes on windows and raises a ValueError if the user
explicitly
Change by Brian Quinlan :
--
assignee: -> bquinlan
___
Python tracker
<https://bugs.python.org/issue26903>
___
___
Python-bugs-list mailing list
Unsubscrib
Brian Quinlan added the comment:
BTW, the 61 process limit comes from:
63 - -
--
___
Python tracker
<https://bugs.python.org/issue26903>
___
___
Python-bug
Brian Quinlan added the comment:
OK, I completely disagree with my statement:
"""If you added this as an argument to shutdown() then you'd probably also have
to add it as an option to the constructors (for people using Executors as
context managers). But, if you h
Change by Brian Quinlan :
--
keywords: +patch
pull_requests: +13045
stage: needs patch -> patch review
___
Python tracker
<https://bugs.python.org/issu
Brian Quinlan added the comment:
Hey Brian, why can't you use threads in your unit tests? Are you worried about
non-determinism or resource usage? Could you make a ThreadPoolExecutor with a
single worker?
--
___
Python tracker
&
Brian Quinlan added the comment:
Can we close this bug then?
--
___
Python tracker
<https://bugs.python.org/issue26374>
___
___
Python-bugs-list mailin
Change by Brian Quinlan :
--
stage: -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue26374>
___
___
Python-bugs-list
Brian Quinlan added the comment:
Do you have a example that you could share?
I can't think of any other fakes in the standard library and I'm hesitant to be
the person who adds the first one ;-)
--
___
Python tracker
<https://bu
Brian Quinlan added the comment:
So you actually use the result of ex.submit i.e. use the resulting future?
If you don't then it might be easier to just create your own thread.
--
___
Python tracker
<https://bugs.python.org/is
Brian Quinlan added the comment:
Using a default executor could be dangerous because it could lead to deadlocks.
For example:
mylib.py
def my_func():
tsubmit(...)
tsubmit(...)
tsubmit(somelib.some_func, ...)
somelib.py
--
def some_func():
tsubmit(...) # Potential
Brian Quinlan added the comment:
Hey Ethan, I'm really sorry about dropping the ball on this. I've been burnt
out on Python stuff for the last couple of years.
When we left this, it looked like the -1s were in the majority and no one new
has jumped on to support `filter`.
If you
Brian Quinlan added the comment:
Hey Hrvoje,
I agree that #1 is the correct approach. `disown` might not be the best name -
maybe `allow_shutdown` or something. But we can bike shed about that later.
Are you interested in writing a patch
Brian Quinlan added the comment:
Hey Brian,
I understand the non-determinism. I was wondering if you had a non-theoretical
example i.e. some case where the non-determinism had impacted a real test that
you wrote?
--
___
Python tracker
<ht
Brian Quinlan added the comment:
If we supported this, aren't we promising that we will always materialize the
iterator passed to map?
I think that we'd need a really strong use-case for this to be worth-while.
--
___
Python track
Brian Quinlan added the comment:
Do the `initializer` and `initargs` parameters deal with this use case for you?
https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
--
___
Python tracker
<ht
Brian Quinlan added the comment:
Ben, do you still think that your patch is relevant or shall we close this bug?
--
___
Python tracker
<https://bugs.python.org/issue22
Brian Quinlan added the comment:
How did the experiment go? Are people still interested in this?
--
___
Python tracker
<https://bugs.python.org/issue22
Change by Brian Quinlan :
--
assignee: -> bquinlan
___
Python tracker
<https://bugs.python.org/issue31783>
___
___
Python-bugs-list mailing list
Unsubscrib
Brian Quinlan added the comment:
Great report Steven!
I was able to reproduce this with the attached patch (just adds some sleeps and
prints) and this script:
from threading import current_thread
from concurrent.futures import ThreadPoolExecutor
from time import sleep
pool
Brian Quinlan added the comment:
I think that ProcessPoolExecutor might have a similar race condition - but not
in exactly this code path since it would only be with the queue management
thread (which is only started once).
--
___
Python tracker
Change by Brian Quinlan :
--
pull_requests: +13087
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue31783>
___
___
Python-bugs-list mai
Brian Quinlan added the comment:
Brian, I was looking for an example where the current executor isn't sufficient
for testing i.e. a useful test that would be difficult to write with a real
executor but would be easier with a fake.
Maybe you have such an example from your
Brian Quinlan added the comment:
Was this fixed by https://github.com/python/cpython/pull/3895 ?
--
___
Python tracker
<https://bugs.python.org/issue30
Brian Quinlan added the comment:
When I first wrote and started using ThreadPoolExecutor, I had a lot of code
like this:
with ThreadPoolExecutor(max_workers=500) as e:
e.map(download, images)
I didn't expect that `images` would be a large list but, if it was, I wanted
all o
Brian Quinlan added the comment:
After playing with it for a while, https://github.com/python/cpython/pull/6375
seems reasonable to me.
It needs tests and some documentation.
Antoine, are you still -1 because of the complexity increase
Brian Quinlan added the comment:
My understanding is that tracebacks have a pretty large memory profile so I'd
rather not keep them alive. Correct me if I'm wrong about that.
--
___
Python tracker
<https://bugs.python.o
Change by Brian Quinlan :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Brian Quinlan added the comment:
We can bike shed over the name in the PR ;-)
--
___
Python tracker
<https://bugs.python.org/issue36780>
___
___
Python-bug
Changes by Brian Quinlan :
--
assignee: -> bquinlan
___
Python tracker
<http://bugs.python.org/issue15015>
___
___
Python-bugs-list mailing list
Unsubscri
Brian Quinlan added the comment:
Hey Nam,
I'm not sure that I understand. You want ThreadPoolExecutor.submit to block if
there are too many work items in the queue? Are you sure that this happens
currently with ProcessPoolExecutor? I can't see wh
Brian Quinlan added the comment:
Thanks for the patch!
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/i
1 - 100 of 152 matches
Mail list logo
