[issue30018] multiprocessing.Pool garbles call stack for __new__

[Charles McEachern]

New submission from Charles McEachern:

I'm calling the constructor of Foo, a subclass of str. Expected output:

Called Foo.__new__ with args = ('TIMESTAMP', 'INPUT0')
TIMESTAMP OUTPUT0

When I make the call using a multiprocessing.pool.ThreadPool, it works fine. 
But when I make the call using a multiprocessing.Pool (using the apply or 
apply_async method), I get:

Called Foo.__new__ with args = ('TIMESTAMP', 'INPUT0')
Called Foo.__new__ with args = ('TIMESTAMP OUTPUT0',)
Exception in thread Thread-3:
...
ValueError: Bad Foo input: ('TIMESTAMP OUTPUT0',)

That is, the object I just constructed seems to be getting shoved right back 
into the constructor. 

When I swap out the Foo class for the similar Goo class, which is not a str, 
and uses __init__ instead of __new__, I again see no problems:

Called Goo.__init__ with args = ('TIMESTAMP', 'INPUT0')


I see this in 2.7.9 as well as 3.4.5. Looks like it's present in 2.7.2 and 
3.5.2 as well:

https://github.com/charles-uno/python-new-pool-bug/issues/1

--
components: Library (Lib)
files: newpool.py
messages: 291278
nosy: Charles McEachern
priority: normal
severity: normal
status: open
title: multiprocessing.Pool garbles call stack for __new__
type: behavior
versions: Python 2.7, Python 3.4, Python 3.5
Added file: http://bugs.python.org/file46790/newpool.py

___
Python tracker 
<http://bugs.python.org/issue30018>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30018] multiprocessing.Pool garbles call stack for __new__

[Charles McEachern]

Charles McEachern added the comment:

This caused me several hours of misery yesterday, trying to isolate what was 
going wrong. 

I am unfortunately not at liberty to share the code I'm working on. The example 
on GitHub has the general thrust of it: my constructor was always called in a 
specific way, and didn't expect to be given something that was already 
processed. 

Interesting to see that this is a product of pickling. That makes me think that 
"fixing" this corner case would probably be a lot of work.

I suppose I should just work around it by checking right away if the input to 
my constructor has already been constructed!

--

___
Python tracker 
<http://bugs.python.org/issue30018>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30018] multiprocessing.Pool garbles call stack for __new__

[Charles McEachern]

Charles McEachern added the comment:

That seems to do it! Looks like the trick is to define __reduce__ to help out 
the serializer. Thanks!

--

___
Python tracker 
<http://bugs.python.org/issue30018>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com