[issue17038] multiprocessing only use one core when C module are imported
New submission from HadiM: Hi, This is the first time I report a bug so if I did something wrong please let me know. I also tried to ask on #python and #python-fr before posting here to know if it was a bug or a possible multiprocessing limitation on Linux. So I'm sorry if it's not a bug... The problem appears when a module with .c compiled file is imported (sush as numpy, scipy, pandas, tables, skimage, I did not tested with a custom .c module). Multiprocessing and more specifically imap and map method of Pool class did not distribute processes along all the core but only on one core. This problem does not appears when I don't import a module with .c compiled. I know distributing processes along cores is specific to the OS implementation and not related to Python but I said that to illustrate the behaviour. I should notice that I did not see the bug with a Windows 7, 32 bits (Virtualbox). My laptop run with the latest ubuntu 64bits (tested with both kernel 3.5 and 3.8 last rc). It's a classic Intel(R) Core(TM) i5-2540M CPU @ 2.60GHz with 4 cores. You can try to reproduce the bug with the attached file. You can also find the same file here : http://pastebin.com/jqq9G5Ph Thank you for your time ! -- components: Library (Lib), ctypes files: bug.py messages: 180648 nosy: hadim priority: normal severity: normal status: open title: multiprocessing only use one core when C module are imported type: behavior versions: Python 2.7 Added file: http://bugs.python.org/file28841/bug.py ___ Python tracker <http://bugs.python.org/issue17038> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17038] multiprocessing only use one core when C module are imported
HadiM added the comment: I test to launch bug.py with pypy (import numpypy instead of import numpy) and the bug did not appear. -- ___ Python tracker <http://bugs.python.org/issue17038> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17038] multiprocessing only use one core when C module are imported
HadiM added the comment: Indeed some value change when I print "cpu" line from /proc/self/status but I don't really understand what that mean... So there is no solution about that ? We cannot use multiprocessing with these modules under Linux ? Do you think I can manually change the CPU affinity or at least block changes made by other modules (without recompiling them of course) ? I guess if it's possible to modify CPU affinity, numpy and other scientific libraries won't be efficient as before, no ? Because the won't share the same cache or something like that, if I get the wikipedia page about cpu affinity. -- ___ Python tracker <http://bugs.python.org/issue17038> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17038] multiprocessing only use one core when C module are imported
HadiM added the comment: Your snippet did the trick ! Thank you for your time. Even if it's not very clean, it's working. Thank again ! -- ___ Python tracker <http://bugs.python.org/issue17038> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
