#include <hallo.h> * Igor Belyi [Wed, Aug 24 2005, 10:52:55AM]: > Eduard Bloch wrote: > > >Interesting. Looks nice, most things are done the same way I did with > >C++, and looking very clean. I was rewritting advfs primarily to make it > >multithreaded, since we have a Dual-Opteron cluster nearby so Klaus > >could compress his DVDs in 5 minutes using best compression rather than > >an hour (or a day for the best compression). The MT codes does already > >work and I have optimised it today for having less locks. Uhm, and I > >changed the processing a bit to work in a similar fashion you do in the > >script, however I dropped the write-to-stdout options because I doubt > >anyone seriosly like to use it. The only users of that way are either > >working with really small images or have lots of RAM plenty, in both > >cases a tmpfs can be used having the same effects. > > > > > Well... The primary idea for me to write the script was to introduce > swapping by the program instead of harassing kernel memory management. > :o) And you need it for read-from-stdin too.
Well, mine too ;) And I see we follow the same ways just in different languages. > >threads could submit the jobs (compression calls) via the xmlrpc objects > >to remote nodes. However, IMO it is a lot of overhead with xmlrpc which > >recodes everything with base64 (afaics looking at its specs) while > >transfering. OTOH it should be possible to write a simple TCP server > >that just sends the data with simple headers... but I am not skilled > >enough write all that in Python. > > > > > I haven't read about Python problems with threads so I went ahead and > introduced threads into the 'create' script: > ftp://katehok.ac93.org/pub/cloop-create-th.py > Use -t option to specify how many threads are dedicated to do the > compression (3 by default) Nice. I have read that paper http://heather.cs.ucdavis.edu/~matloff/Python/PyThreads.pdf to get an overview on its features. > I don't have SMP machine so I can't verify if it benefits from multiple > processors but I'd be glad to hear about your results if you want to try I tested it, and it is really faster: real 1m7.000s user 3m21.535s sys 0m1.547s vs. real 1m44.441s user 1m42.108s sys 0m2.251s However it is not that fast as with native system threads that I used: real 0m53.302s user 1m42.593s sys 0m1.412s OTOH on a machine with kernel 2.4.21 and 2 Opterons the threaded script and threaded C code have almost identical speed, Python is just few percents slower. I don't think that different comes from sligthly different locking schemes, though. Regards, Eduard. -- "The quiet ones are the ones who change the universe. The loud ones only take the credit." L.M. ItB -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
