[issue33166] os.cpu_count() returns wrong number of processors on specific systems
Change by yanir hainick : -- components: Windows nosy: paul.moore, steve.dower, tim.golden, yanirh, zach.ware priority: normal severity: normal status: open type: behavior ___ Python tracker <https://bugs.python.org/issue33166> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33166] os.cpu_count() returns wrong number of processors on specific systems
yanir hainick added the comment: wrong number of cpu's is reported on some specific platforms. *** first platform: server with X4 Intel Xeon E5-4620 (8 physical, 16 logical), running a 64bit Windows Server 2012 R2 Standard. results: os.cpu_count() reports 64 units psutil.cpu_count(logical=False) reports 32 units psutil.cpu_count(logical=True) reports 64 units multiprocessing using concurrent.futures able to fully utilize the server; *** second platform: server with X2 Intel Xeon Gold 6138 (20 physical, 40 logical), running a 64bit Windows Server 2016 Standard. results: os.cpu_count() reports 128 units psutil.cpu_count(logical=False) reports 20 units psutil.cpu_count(logical=True) reports 40 units multiprocessing using concurrent.futures able to utilize only 1/4 of the server's power; -- ___ Python tracker <https://bugs.python.org/issue33166> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33166] os.cpu_count() returns wrong number of processors on specific systems
yanir hainick added the comment: Yup. Attaching a screenshot. -- Added file: https://bugs.python.org/file47504/Screenshot.png ___ Python tracker <https://bugs.python.org/issue33166> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33166] os.cpu_count() returns wrong number of processors on specific systems
yanir hainick added the comment: Maybe i'm missing something, and would appreciate clarification. Perhaps psutil is wrong, but it gives an answer that has something to do with the actual situation. On platform 2, i have 2 Intel Xeon Gold 6138, each with 20 physical processors, 40 logicals. you are saying i need to rely on os.cpu_count(), which outputs '128'. Can you elaborate on this? Moreover, when attempting to parallelize on the processors, i reach 25% utilization, which suggests Python 'sees' only one processor group. -- ___ Python tracker <https://bugs.python.org/issue33166> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33166] os.cpu_count() returns wrong number of processors on specific systems
yanir hainick added the comment: Yes. Both are wrong, and os.cpu_count() is completely off. Regarding how to determine the number of physical/logical cores in my machine - well, not sure what you mean by that. I've attached a screenshot of Windows' system information. Also used 'cpu-z'. It seems like aside from the os.cpu_count() issue, Python itself has some problem - it 'sees' only 1 CPU group. It is evident from the fact the when parallelizing, utilization level is only 25%. -- ___ Python tracker <https://bugs.python.org/issue33166> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33166] os.cpu_count() returns wrong number of processors on specific systems
yanir hainick added the comment: Ok, no problem. Just to be sure i'm doing the right thing - this thread will be dedicated to the os.cpu_count() issue, and i'll open a new issue on the parallelization problem. makes sense? -- ___ Python tracker <https://bugs.python.org/issue33166> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33171] multiprocessing won't utilize all of platform resources
New submission from yanir hainick : I'm using either multiprocessing package or concurrent.futures for some embarrassingly parallel application. I performed a simple test: basically making n_jobs calls for a simple function - 'sum(list(range(n)))', with n large enough so that the operation is a few seconds long - where n_jobs > n_logical_cores. Tried it on two platforms: first platform: server with X4 Intel Xeon E5-4620 (8 physical, 16 logical), running a 64bit Windows Server 2012 R2 Standard. *** second platform: server with X2 Intel Xeon Gold 6138 (20 physical, 40 logical), running a 64bit Windows Server 2016 Standard. *** first platform reaches 100% utilization. second platform reaches 25% utilization. -- components: Windows messages: 314600 nosy: paul.moore, steve.dower, tim.golden, yanirh, zach.ware priority: normal severity: normal status: open title: multiprocessing won't utilize all of platform resources type: behavior versions: Python 3.6 ___ Python tracker <https://bugs.python.org/issue33171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
