#include <hallo.h>
* Igor Belyi [Wed, Aug 24 2005, 10:52:55AM]:
> Eduard Bloch wrote:
> 
> >Interesting. Looks nice, most things are done the same way I did with
> >C++, and looking very clean. I was rewritting advfs primarily to make it
> >multithreaded, since we have a Dual-Opteron cluster nearby so Klaus
> >could compress his DVDs in 5 minutes using best compression rather than
> >an hour (or a day for the best compression). The MT codes does already
> >work and I have optimised it today for having less locks. Uhm, and I
> >changed the processing a bit to work in a similar fashion you do in the
> >script, however I dropped the write-to-stdout options because I doubt
> >anyone seriosly like to use it. The only users of that way are either
> >working with really small images or have lots of RAM plenty, in both
> >cases a tmpfs can be used having the same effects.
> > 
> >
> Well... The primary idea for me to write the script was to introduce 
> swapping by the program instead of harassing kernel memory management. 
> :o) And you need it for read-from-stdin too.

Well, mine too ;) And I see we follow the same ways just in different
languages.

> >threads could submit the jobs (compression calls) via the xmlrpc objects
> >to remote nodes. However, IMO it is a lot of overhead with xmlrpc which
> >recodes everything with base64 (afaics looking at its specs) while
> >transfering. OTOH it should be possible to write a simple TCP server
> >that just sends the data with simple headers... but I am not skilled
> >enough write all that in Python.
> > 
> >
> I haven't read about Python problems with threads so I went ahead and 
> introduced threads into the 'create' script:
> ftp://katehok.ac93.org/pub/cloop-create-th.py
> Use -t option to specify how many threads are dedicated to do the 
> compression (3 by default)

Nice. I have read that paper
http://heather.cs.ucdavis.edu/~matloff/Python/PyThreads.pdf to get an
overview on its features.

> I don't have SMP machine so I can't verify if it benefits from multiple 
> processors but I'd be glad to hear about your results if you want to try 

I tested it, and it is really faster:

real    1m7.000s
user    3m21.535s
sys     0m1.547s

vs.

real    1m44.441s
user    1m42.108s
sys     0m2.251s

However it is not that fast as with native system threads that I used:

real    0m53.302s
user    1m42.593s
sys     0m1.412s

OTOH on a machine with kernel 2.4.21 and 2 Opterons the threaded script and
threaded C code have almost identical speed, Python is just few percents
slower. I don't think that different comes from sligthly different
locking schemes, though.

Regards,
Eduard.
-- 
"The quiet ones are the ones who change the universe.
 The loud ones only take the credit."
                                                               L.M. ItB


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Reply via email to