On 12/16/2010 02:29 PM, Michael Foord wrote:
On 16/12/2010 11:09, Dimitrios Pritsos wrote:
Hello Core Developers,
My name is Dimitrios and I am newbie in python. I am working on a
Project (part of my PhD) that is called Synergeticprocessing module.
Initially is imitating the multiprocessing built in module but the
processes are distributed on a LAN and not Locally. The main issue I
have is with Pickle module. And I think I found some kind of BUG in
the built in multiprocessing module.
Hello Dimitrios,
Please post your bug report to the Python bug tracker. As you think
you have a fix for the issue it is much more likely to be applied
quickly if you can package it in the form of a test that demonstrates
the issue and a patch that fixes it.
http://bugs.python.org/
All the best,
Michael Foord
Hello Michael,
OK I will do sent it to the bug tracker. But what about the last issue
i.e. that even if the class is transfered-and-pickled-unpickled it
raises an exception if the class is defined into the __main__ script and
not imported from an other one. (see below) could you please guide me
where should I post this for someone to help me fix it!
Best Regards,
Dimitrios
(Synergeticprocessing module is located at GitHub:
https://github.com/dpritsos/synergeticprocessing)
Starting with the "BUG". In case someone uses the
multiprocessing.Pool of processes he/she has to face the problem of
types.MehtodType Impossible pickling. That is you cannot dispatch an
class instance method to the to the Process Pool. However, digging in
to the Source Code of the module there are few lines that resolve
this issue however are not activated or they are faultily activated
so they do not work. This is the 'BUG'
_@ ....../multiprocessing/forking.py_
.
.
.
#
# Try making some callable types picklable
#
from pickle import Pickler
class ForkingPickler(Pickler):
dispatch = Pickler.dispatch.copy()
@classmethod
def register(cls, type, reduce):
def dispatcher(self, obj):
rv = reduce(obj)
self.save_reduce(obj=obj, *rv)
cls.dispatch[type] = dispatcher
def _reduce_method(m):
if m.im_self is None:
return getattr, (m.im_class, m.im_func.func_name)
else:
return getattr, (m.im_self, m.im_func.func_name)
ForkingPickler.register(type(ForkingPickler.save), _reduce_method)
def _reduce_method_descriptor(m):
return getattr, (m.__objclass__, m.__name__)
ForkingPickler.register(type(list.append), _reduce_method_descriptor)
ForkingPickler.register(type(int.__add__), _reduce_method_descriptor)
#def _reduce_builtin_function_or_method(m):
# return getattr, (m.__self__, m.__name__)
#ForkingPickler.register(type(list().append),
_reduce_builtin_function_or_method)
#ForkingPickler.register(type(int().__add__),
_reduce_builtin_function_or_method)
.
.
.
The RED lines are not doing the job, for some reason they are not
managing to register the GREEN function as a global reduce/pickling
function even if you call the registration function into you __main__
script.
The solution I found is just to do this
*
import copy_reg
import types*
def _reduce_method(m):
if m.im_self is None:
return getattr, (m.im_class, m.im_func.func_name)
else:
return getattr, (m.im_self, m.im_func.func_name)
*copy_reg.pickle(types.MethodType, _reduce_method)*
.
.
.
Doing that everything works FINE. But ONLY for local methods i.e. the
ones that their class is defined on the __main__ script or other
import-ed.
In case you want to send something remotely (in an other machine) or
to an other __main__ script running separately then you get a message
like this:
'module' object has no attribute '<my_class>'
The only way to resolve this is firstly to import a script that has
<my_class> defined there and everything works fine.
SO the problems it seems to be that the *m.im_class* (look code
above) has some attribute __module__ defined as __module__ =
'__main__' or something like that. And this is the reason why remote
script cannot execute the function. I mean that the _reduce_method()
above DOES is pickling the whole CLASS object so there is no reason
not to be executed at the remote script. Besides it does as mentioned
above in you just import this the user defined class form an other
script.
I have already spent about 12 weeks working on building my
synergeticPool and resolve the issue of Pickling and only 2 days
needed for the code of the Pool and the rest of the time was spent
for the Pickling issues, and study all the Class related mechanics of
python. That was the reason I ve started digging the
multipocessessing module and found this say 'BUG', and finally sent
this email.
Best Regards,
Dimitrios
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
--
http://www.voidspace.org.uk/
May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessinghttp://www.sqlite.org/different.html
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
