[issue34034] Python 3.7.0 multiprocessing forkserver ForkingPickler behaviour change

2018-07-03 Thread Santiago Hernandez 


New submission from Santiago Hernandez :

Hi,

I am splitting some processing among different processes with the forkserver 
start method using multiprocessing.
In python 3.4 to 3.6 this worked fine.
In python 3.7.0 it's breaking with the following Traceback:

```
  File 
"/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/process.py", line 
112, in start
self._popen = self._Popen(self)
  File 
"/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/context.py", line 
223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
  File 
"/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/context.py", line 
284, in _Popen
return Popen(process_obj)
  File 
"/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/popen_spawn_posix.py",
 line 32, in __init__
super().__init__(process_obj)
  File 
"/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/popen_fork.py", 
line 20, in __init__
self._launch(process_obj)
  File 
"/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/popen_spawn_posix.py",
 line 47, in _launch
reduction.dump(process_obj, fp)
  File 
"/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/reduction.py", 
line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle weakref objects
```

I am attaching a minimum example where this happens.
In the example this can be solved by making self._run_single a staticmethod.

However, I would like to understand what is going on and what changed in python 
for this to throw an exception now, when before it did not.

Thanks.
Santiago

--
files: example.py
messages: 320995
nosy: Santiago Hernandez
priority: normal
severity: normal
status: open
title: Python 3.7.0 multiprocessing forkserver ForkingPickler behaviour change
versions: Python 3.7
Added file: https://bugs.python.org/file47670/example.py

___
Python tracker 
<https://bugs.python.org/issue34034>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34034] Python 3.7.0 multiprocessing forkserver ForkingPickler behaviour change

2018-07-06 Thread Santiago Hernandez 


Santiago Hernandez  added the comment:

Thanks, I get it now, I hadn't seen that the pickling of the workers was
Processes pickling.
Yeah, I was not doing this anywhere else, it seems like slipped through
once. But it called my attention that it used to work and now it doesn't.

On Tue, Jul 3, 2018 at 4:38 PM, Antoine Pitrou 
wrote:

>
> Antoine Pitrou  added the comment:
>
> It could be many different things.  The bottom line here, though, is that
> the Process class is not designed to be picklable (how would it work?),
> which is probably why you're seeing this.
>
> --
> nosy: +pitrou
>
> ___
> Python tracker 
> <https://bugs.python.org/issue34034>
> ___
>

--

___
Python tracker 
<https://bugs.python.org/issue34034>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com