Camaleón put forth on 1/11/2011 9:38 AM: > I supposed you wouldn't care much in getting a script to run faster with > all the available core "occupied" if you had a modern (<4 years) cpu and > plenty of speedy ram because the routine you wanted to run it should not > take many time... unless you were going to process "thousand" of > images :-)
That's a bit ironic. You're suggesting the solution is to upgrade to a new system with a faster processor and memory. However, all the newer processors have 2, 4, 6, 8, or 12 cores. So upgrading simply for single process throughput would waste all the other cores, which was the exact situation I found myself in. The ironic part is that parallelizing the script to maximize performance on my system will also do the same for the newer chips, but to an even greater degree on those with 4, 6, 8, or 12 cores. Due to the fact that convert doesn't eat 100% of a core's time during its run, and the idle time in between one process finishing and xargs starting another, one could probably run 16-18 parallel convert processes on a 12 core Magny Cours with this script before run times stop decreasing. The script works. It cut my run time by over 50%. I'm happy. As I said, this system's processing power is complete overkill 99% of the time. It works beautifully with pretty much everything I've thrown at it, for 8 years now. If I _really_ wanted to maximize the speed of this photo resizing task I'd install Win32 ImageMagick on my 2GHz Athlon XP workstation with dual channel memory nForce2 mobo, convert them on the workstation, and copy them to the server. However, absolute maximum performance of this task was not, and is not my goal. My goal was to make use of the second CPU, which was sitting idle in the server, to speed up the task completion. That goal was accomplished. :) >>> Running more processes than real cores seems fine, did you try it? >> >> Define "fine". > > Fine = system not hogging all resources. I had run 4 (2 core machine) and run time was a few seconds faster than 2 processes, 3 seconds IIRC. Running 8 processes pushed the system into swap and run time increased dramatically. Given that 4 processes only have a few seconds faster than two, yet consumed twice as much memory, the best overall number of processes to run on this system is two. > I didn't know the meaning of that "SUT" term... I like using it. It's good short hand. I wish more people used it, or were familiar with it, so I wouldn't have to define it every time I use it. :) > The test was run in a > laptop (Toshiba Tecra A7) with an Intel Core Duo T2400 (in brief, 2M > Cache, 1.83 GHz, 667 MHz FSB, full specs¹) and 4 GiB of ram (DDR2) > VM is Virtualbox (4.0) with Windows XP Pro as host and Debian Squeeze as > guest. VM was setup to use the 2 cores and 1.5 GiB of system ram. Disk > controller is emulated via ich6. I wonder how much faster convert it would run on bare metal on that laptop. >> Are you "new" to the concept of parallel processing and what CPU process >> scheduling is? > > No... I guess this is quite similar to the way most of the daemons do > when running in background and launch several instances (like "amavisd- > new" does) but I didn't think there was a direct relation in the number > of the running daemons/processes and the cores available in the CPU, I > mean, I thought the kernel would automatically handle all the resources > available the best it can, regardless of the number of cores in use. This is correct. But the kernel can't take a single process make it run across all cores, maximizing performance. For this, the process must be written to create threads, forks, or children. The kernel will then run each of these on a different processor core. This is why Imagemagick convert needs to be parallelized when batching many photos. If you don't parallelize it, the kernel can't schedule it across all cores. The docs say it will use threads but only with "large" files. Apparently 8.2 megapixel JPGs aren't "large", as the threading has never kicked in for me. By using xargs for parallelization, we create x number of concurrent processes. The kernel then schedules each one on a different cpu core. -- Stan -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4d2cd295.30...@hardwarefreak.com
