[issue24900] Raising an exception that is unable to be unpickled causes hang in ProcessPoolExecutor

2015-08-20 Thread Benedikt Sauer 

New submission from Benedikt Sauer:

When raising an exception that is unpicklable in a worker process. This used to 
be the case for all exceptions that have a non-trivial constructor but this was 
fixed by http://bugs.python.org/issue1692335.

I now have the concrete problem with cx_Oracle in which the error type 
cx_Oracle._Error doesn't define tp_new which results in a

UnpicklingError: NEWOBJ class argument has NULL tp_new

I have already reported this fact at 
https://bitbucket.org/anthony_tuininga/cx_oracle/issues/23.

However, I think that the worker thread should either be able to recover from 
this (which should be quite easy) or, at the very least, fail cleanly. 
Currently, when this condition is triggered the worker processes are not killed 
alongside the parent.

There is an existing report and a patch in the backports project: 
https://github.com/agronholm/pythonfutures/issues/30#issuecomment-132576545

--
messages: 248878
nosy: filmor
priority: normal
severity: normal
status: open
title: Raising an exception that is unable to be unpickled causes hang in 
ProcessPoolExecutor
versions: Python 3.3

___
Python tracker 
<http://bugs.python.org/issue24900>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24900] Raising an exception that cannot be unpickled causes hang in ProcessPoolExecutor

2015-08-20 Thread Benedikt Sauer 

Benedikt Sauer added the comment:

Actually I think that this may consist of multiple issues:

1. http://bugs.python.org/issue8323 (closed as wont-fix): multiprocessing.Queue 
should throw a sensible error if a Queue element can't be unpickled, however 
the user managed to create that
2. The worker process should prevent unpicklable ResultItems to be put on the 
Queue
3. The parent thread in ProcessPoolExecutor should handle the exceptions by 
setting its broken flag and at least trying to cleanly shut down

--
title: Raising an exception that is unable to be unpickled causes hang in 
ProcessPoolExecutor -> Raising an exception that cannot be unpickled causes 
hang in ProcessPoolExecutor

___
Python tracker 
<http://bugs.python.org/issue24900>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com