Re: How to limit CPU usage in Python
On Jueves septiembre 20 2012 11:12:44 Rolando Cañer Roblejo escribió: > Hi all, > > Is it possible for me to put a limit in the amount of processor usage (% > CPU) that my current python script is using? Is there any module useful > for this task? I saw Resource module but I think it is not the module I > am looking for. Some people recommend to use nice and cpulimit unix > tools, but those are external to python and I prefer a python solution. > I am working with Linux (Ubuntu 10.04). > > Best regards. Hola, Sometimes a stupid solution like the following does the trick: > import time > for t in tasks: > do_something(t) > time.sleep(some_seconds) where "some_seconds" is a number related to the typical time-scale of the tasks you are doing. Hope it helps, Regards -- Miller's Slogan: Lose a few, lose a few. -- http://mail.python.org/mailman/listinfo/python-list
Re: Binding first parameter of method to constant value
On Viernes mayo 18 2012 14:14:33 Johannes Bauer escribió: > Hi group, > > I'm trying to dynamically add methods to a class at runtime. What I > would like to do is have them all delegated to one common function which > is called with the name of the function as first parameter. I.e.: > > class Foo(): > def __init__(self): > # Some magic missing here > setattr(self, "foometh", types.MethodType(self._dispatcher, self)) > setattr(self, "barmeth", types.MethodType(self._dispatcher, self)) > > def _dispatcher(self, caller, *args): > # Dispatcher called with caller == "foometh" > # or caller == "barmeth", depending on which one was called > > > with the effect that > > f = Foo() > f.foometh(1, 2, 3) > # -> this should be equivaent to Foo._dispatcher(f, "foometh", 1, 2, 3) > > f.barmeth() > # -> this should be equivaent to Foo._dispatcher(f, "barmeth") > > I'm kind of stuck. Can you please give me hints? > > Best regards, > Joe > > >> Wo hattest Du das Beben nochmal GENAU vorhergesagt? > > > > Zumindest nicht öffentlich! > > Ah, der neueste und bis heute genialste Streich unsere großen > Kosmologen: Die Geheim-Vorhersage. > - Karl Kaos über Rüdiger Thomas in dsa Hi Johannes, If I understood well your question, for specific attributes I recommend you to have a look at properties and descriptors (I love them). For a general way to deal with attributes: the __getattr__ (to handle undefined attributes), __getattribute__ (to handle *every* attribute; be careful with loops), and __setattr__ (to catch *every* attribute assignmet, same warning concerning loops). Hope it helps. BR, DPalao -- http://mail.python.org/mailman/listinfo/python-list
70% [* SPAM *] multiprocessing.Queue blocks when sending large object
Hello, I'm trying to use multiprocessing to parallelize a code. There is a number of tasks (usually 12) that can be run independently. Each task produces a numpy array, and at the end, those arrays must be combined. I implemented this using Queues (multiprocessing.Queue): one for input and another for output. But the code blocks. And it must be related to the size of the item I put on the Queue: if I put a small array, the code works well; if the array is realistically large (in my case if can vary from 160kB to 1MB), the code blocks apparently forever. I have tried this: http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and-large- objects/ but it didn't work (especifically I put a None sentinel at the end for each worker). Before I change the implementation, is there a way to bypass this problem with multiprocessing.Queue? Should I post the code (or a sketchy version of it)? TIA, David -- http://mail.python.org/mailman/listinfo/python-list
70% [* SPAM *] Re: multiprocessing.Queue blocks when sending large object
El Martes Noviembre 29 2011, DPalao escribió: > Hello, > I'm trying to use multiprocessing to parallelize a code. There is a number > of tasks (usually 12) that can be run independently. Each task produces a > numpy array, and at the end, those arrays must be combined. > I implemented this using Queues (multiprocessing.Queue): one for input and > another for output. > But the code blocks. And it must be related to the size of the item I put > on the Queue: if I put a small array, the code works well; if the array is > realistically large (in my case if can vary from 160kB to 1MB), the code > blocks apparently forever. > I have tried this: > http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and-lar > ge- objects/ > but it didn't work (especifically I put a None sentinel at the end for each > worker). > > Before I change the implementation, > is there a way to bypass this problem with multiprocessing.Queue? > Should I post the code (or a sketchy version of it)? > > TIA, > > David Just for reference. The other day I found the explanation by "ryles" on his/her mail of 27th aug 2009, with title "Re: Q: multiprocessing.Queue size limitations or bug...". It is very clarifying. After having read that I arranged the program such that the main process did not need to know when the others have finished, so I changed the process join call with a queue get call, until a None (one per process) is returned. Best, David -- http://mail.python.org/mailman/listinfo/python-list
70% [* SPAM *] Re: Re: multiprocessing.Queue blocks when sending large object
El Lunes Diciembre 5 2011, [email protected] escribió: > On Mon, 5 Dec 2011 09:02:08 +0100, DPalao > > wrote: > >El Martes Noviembre 29 2011, DPalao escribió: > >> Hello, > >> I'm trying to use multiprocessing to parallelize a code. There is a > >> number of tasks (usually 12) that can be run independently. Each task > >> produces a numpy array, and at the end, those arrays must be combined. > >> I implemented this using Queues (multiprocessing.Queue): one for input > >> and another for output. > >> But the code blocks. And it must be related to the size of the item I > >> put on the Queue: if I put a small array, the code works well; if the > >> array is realistically large (in my case if can vary from 160kB to > >> 1MB), the code blocks apparently forever. > >> I have tried this: > >> http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and- > >> lar ge- objects/ > >> but it didn't work (especifically I put a None sentinel at the end for > >> each worker). > >> > >> Before I change the implementation, > >> is there a way to bypass this problem with multiprocessing.Queue? > >> Should I post the code (or a sketchy version of it)? > >> > >> TIA, > >> > >> David > > > >Just for reference. The other day I found the explanation by "ryles" on > >his/her mail of 27th aug 2009, with title "Re: Q: multiprocessing.Queue > >size limitations or bug...". It is very clarifying. > >After having read that I arranged the program such that the main process > >did not need to know when the others have finished, so I changed the > >process join call with a queue get call, until a None (one per process) > >is returned. > > > >Best, > > > >David > > Why do people add character like[* SPAM *] to their subject > lines ?? Is it supposed to do something ?? I figured since > programmers hang out here, maybe one of you know this. > > Thanks, > boB Obviously it was not me who added the disgusting "70% [* SPAM *]" string to the subject. And I'd like to know the answer too. David -- http://mail.python.org/mailman/listinfo/python-list
70% [* SPAM *] Re: multiprocessing.Queue blocks when sending large object
Hi Lie, Thank you for the reply. El Lunes Diciembre 5 2011, Lie Ryan escribió: > On 11/30/2011 06:09 AM, DPalao wrote: > > Hello, > > I'm trying to use multiprocessing to parallelize a code. There is a > > number of tasks (usually 12) that can be run independently. Each task > > produces a numpy array, and at the end, those arrays must be combined. > > I implemented this using Queues (multiprocessing.Queue): one for input > > and another for output. > > But the code blocks. And it must be related to the size of the item I put > > on the Queue: if I put a small array, the code works well; if the array > > is realistically large (in my case if can vary from 160kB to 1MB), the > > code blocks apparently forever. > > I have tried this: > > http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and-l > > arge- objects/ > > but it didn't work (especifically I put a None sentinel at the end for > > each worker). > > > > Before I change the implementation, > > is there a way to bypass this problem with multiprocessing.Queue? > > Should I post the code (or a sketchy version of it)? > > Transferring data over multiprocessing.Queue involves copying the whole > object across an inter-process pipe, so you need to have a reasonably > large workload in the processes to justify the cost of the copying to > benefit from running the workload in parallel. > > You may try to avoid the cost of copying by using shared memory > (http://docs.python.org/library/multiprocessing.html#sharing-state-between- > processes); you can use Queue for communicating when a new data comes in or > when a task is done, but put the large data in shared memory. Be careful > not to access the data from multiple processes concurrently. > Yep, that was my first thought, but the arrays's elements are complex64 (or complex in general), and I don't know how to easily convert from multiprocessing.Array to/from numpy.array when the type is complex. Doing that would require some extra conversions forth and back which make the solution not very attractive to me. I tried with a Manager too, but the array cannot be modified from within the worker processes. In principle, the array I need to share is expected to be, at most, ~2MB in size, and typically should be only <200kB. So, in principle, there is no huge extra workload. But that could change, and I'd like to be prepared for it, so any idea about using an Array or a Manager or another shared memory thing would be great. > In any case, have you tried a multithreaded solution? numpy is a C > extension, and I believe it releases the GIL when working, so it > wouldn't be in your way to achieve parallelism. That possibility I didn't know. What does exactly break the GIL? The sharing of a numpy array? What if I need to also share some other "standard" python data (eg, a dictionary)? Best regards, David -- http://mail.python.org/mailman/listinfo/python-list
70% [* SPAM *] shelve trying to import bsddb
Dear all, I'm trying to use shelve to store some data, but sheve itself tries to import bsddb, which results in: > File "/usr/lib64/python2.6/shelve.py", line 239, in open > return DbfilenameShelf(filename, flag, protocol, writeback) > File "/usr/lib64/python2.6/shelve.py", line 223, in __init__ > Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback) > File "/usr/lib64/python2.6/anydbm.py", line 82, in open > mod = __import__(result) > File "/usr/lib64/python2.6/dbhash.py", line 8, in > import bsddb > ImportError: No module named bsddb I know that the new bsddb3 should be used, so what can I do? TIA, D -- http://mail.python.org/mailman/listinfo/python-list
70% [* SPAM *] Re: shelve trying to import bsddb
El Wednesday February 16 2011, DPalao escribió: > Dear all, > I'm trying to use shelve to store some data, but sheve itself tries to > import > > bsddb, which results in: > > File "/usr/lib64/python2.6/shelve.py", line 239, in open > > > > return DbfilenameShelf(filename, flag, protocol, writeback) > > > > File "/usr/lib64/python2.6/shelve.py", line 223, in __init__ > > > > Shelf.__init__(self, anydbm.open(filename, flag), protocol, > > writeback) > > > > File "/usr/lib64/python2.6/anydbm.py", line 82, in open > > > > mod = __import__(result) > > > > File "/usr/lib64/python2.6/dbhash.py", line 8, in > > > > import bsddb > > > > ImportError: No module named bsddb > > I know that the new bsddb3 should be used, so what can I do? > TIA, > > D I solved the issue. I just compiled python (btw, I'm using 2.6.6) with support for "berkdb", which apparently disappeared in a recent update (I'm using gentoo). -- http://mail.python.org/mailman/listinfo/python-list
