I am a beginner with Python, coming from R, and I am having problems with
parallelization with the multiprocessing module. I know that other people have
asked similar questions but the answers were mostly over my head.
Here is my problem: I tried to execute code in parallel in two ways:
1) In a plain xyz.py file without calling main()
2) In a xyz.py file that calls main
Under 1) I was able to run parallel processes but:
a) The whole script runs from the beginning up to the line where p1.start() or
p2.start() is called. That is, if I had 10 processes p1, p2, ..., p10 the whole
file would be run from the beginning up to the line where the command
pX.start() is called. Maybe it has to be that way so that these processes get
the environment they need, but I doubt it.
b) I was not able to extract a value from the function called. I was able only
to use print(). I tried to create a Queue object to get the return values but
then I get error messages:
> from multiprocessing import *
>
> print('\nRunning ' + __name__ + "\n")
>
> from multiprocessing import Process, Queue, freeze_support
> freeze_support() # it does not make any difference to run this command
> or not
>
> queue1 = Queue() # create a queue object
>
>
> def multiply(a, b, que):
> que.put(a * b)
>
> p = Process(target=multiply, args=(5, 4, queue1))
> p.start()
> p.join()
Traceback (most recent call last):
File "", line 1, in
File
"C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py",
line 350, in main
prepare(preparation_data)
File
"C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py",
line 457, in prepare
'__parents_main__', file, path_name, etc
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\imp.py", line 175, in
load_module
return load_source(name, filename, file)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\imp.py", line 114, in
load_source
_LoadSourceCompatibility(name, pathname, file).load_module(name)
File "", line 586, in _check_name_wrapper
File "", line 1024, in load_module
File "", line 1005, in load_module
File "", line 562, in module_for_loader_wrapper
File "", line 870, in _load_module
File "", line 313, in _call_with_frames_removed
File "C:\Files\Programs\Wush\Python\parallel_test_2.py", line 15, in
p.start()
File
"C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\process.py",
line 111, in start
self._popen = Popen(self)
File
"C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py",
line 216, in __init__
cmd = ' '.join('"%s"' % x for x in get_command_line())
File
"C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py",
line 328, in get_command_line
is not going to be frozen to produce a Windows executable.''')
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.
This probably means that you are on Windows and you have
forgotten to use the proper idiom in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce a Windows executable.
>>>
Under 2) I get problems with pickling. See below
from multiprocessing import *
def main():
print('\nRunning ' + __name__ + "\n")
freeze_support()
def f(name):
print('hello', name)
p = Process(target=f, args=('bob',))
p.start()
p.join()
if __name__ == '__main__':
main()
Running __main__
Traceback (most recent call last):
File "", line 1, in
File
"C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py",
line 562, in runfile
execfile(filename, namespace)
File
"C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py",
line 41, in execfile
exec(compile(open(filename).read(), filename, 'exec'), namespace)
File "C:\Files\Programs\Wush\Python\parallel_test.py", line 18, in
main()
File "C:\Files\Programs\Wush\Python\parallel_test.py", line 14, in main
p.start()
File
"C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\process.py",
line 111, in start
self._popen = Popen(self)
File
"C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py",
line 248, in __init__
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File
"C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\multiprocessing\forking.py",
line 166, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't
