----- "Mark Kosmowski" <[EMAIL PROTECTED]> wrote: > This discussion as well as reading about NUMA and affinity elsewhere > leads to another question - what is the difference between using > numactl or using the affinity options of my parallelization software > (in my case openmpi)?
Our experiences with MVAPICH2 was unpleasant because of our workload (lots of users with their own codes doing very different - and occasionally wrong - things). It has a naive implementation where if you have say an 8 way system and run 2 x 4-way MPI jobs on it then this happens: Job 1 starts and each MPI thread sets affinity starting from core 0, so it allocates core 0, 1, 2 and 3. Job 2 starts and each MPI thread sets affinity starting from core 0, so it allocates core 0, 1, 2 and 3. This is a Bad Thing(tm). Suddenly the user wonders why their performance has just halved. The sysadmin looks at the node and wonders what the code is doing wrong if the load average is so high but there's so much idle CPU time available. This is why I'm a big fan of the queueing system doing this for you, especially if it can (as Torque does when you use a compatible MPI job launcher) allocate each MPI process onto its own core. Of course it's game over if you're using a legacy MPI launcher that uses ssh or rsh. :-( cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
