Eduard Bloch wrote:
Interesting. Looks nice, most things are done the same way I did with
C++, and looking very clean. I was rewritting advfs primarily to make it
multithreaded, since we have a Dual-Opteron cluster nearby so Klaus
could compress his DVDs in 5 minutes using best compression rather than
an hour (or a day for the best compression). The MT codes does already
work and I have optimised it today for having less locks. Uhm, and I
changed the processing a bit to work in a similar fashion you do in the
script, however I dropped the write-to-stdout options because I doubt
anyone seriosly like to use it. The only users of that way are either
working with really small images or have lots of RAM plenty, in both
cases a tmpfs can be used having the same effects.
Well... The primary idea for me to write the script was to introduce
swapping by the program instead of harassing kernel memory management.
:o) And you need it for read-from-stdin too.
As said, your code looked nice and for a (long) moment I even though
about continueing my work with Python based on your code but then I have
read that Python does not work with kernel threads and basically does no
benefit from SMP. Okay, that is not that bad, you can even work with one
thread only and pass the jobs over network. IMO one could do
multithreading with Python Queue objects for passing the jobs between
the reader thread, worker threads and output thread. And the worker
threads could submit the jobs (compression calls) via the xmlrpc objects
to remote nodes. However, IMO it is a lot of overhead with xmlrpc which
recodes everything with base64 (afaics looking at its specs) while
transfering. OTOH it should be possible to write a simple TCP server
that just sends the data with simple headers... but I am not skilled
enough write all that in Python.
I haven't read about Python problems with threads so I went ahead and
introduced threads into the 'create' script:
ftp://katehok.ac93.org/pub/cloop-create-th.py
Use -t option to specify how many threads are dedicated to do the
compression (3 by default)
I don't have SMP machine so I can't verify if it benefits from multiple
processors but I'd be glad to hear about your results if you want to try
it out.
Cheers,
Igor
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
