Hey Stuart, Thanks much for the detail.
So, if I am reading you correctly your test was on a single physical PHI (you will later expand to multiple PHIs). This was a highly parallel single precision application which showed the expected linear speed up to 60 cores ... then a kink as you cross into hyper-threaded operation with a 1/2 as steep slope up to factor of two to 120 core-equivalents with a 4 to 1 over subscription of hyper-threads. This was all done with the Intel compilers on an unmodified pthreaded code that is well-vectored. A good result ... on an application that is a perfect candidate for PHI. To run elsewhere with CUDA, OpenMP, or OpenACC directives would require quite a bit of recoding which you were happy to avoid. My guess is if you had a CUDA implementation you would see better performance on a FERMI or KEPLER, but that is a programming path you do not wish to take. This is an interesting case to hear about. The flack (technical marketing) from NVIDIA is to focus on the difficulty of using the 'offload' model and Intel extensions to OpenMP, Cylk, etc., articulate their hardware's performance advantages, and talk about OpenACC. These arguments are not unreasonable, but clearly not universallydeciding. Thanks much ... and good luck getting all your other codes to scale just as well. rbw On Thu, Feb 14, 2013 at 10:18 AM, Dr Stuart Midgley <sdm...@gmail.com>wrote: > Evening > > Sorry for the slow response. > > Most of our codes are pthreads, we have avoided MPI and OpenMP as much as > possible. Our current cluster consists of Nehalem, Westmere, Sandy Bridge > and Interlagos of various flavours. Our Phi cards are in Sandy Bridge > systems (host machine has 16 cores with 128GB ram). We run the intel > compilers. > > Our fastest systems are the 64core Interlagos systems (256GB ram) running > at 2.6GHz. For a few of our most important kernels, a single phi had > greater throughput than a whole node. Which, if you count the flops, is > expected. The Phi's have a massive amount of single precision floating > point performance (our codes are single precision). > > Our kernels vectorise very well (lots of hand coded SSE3) and are expected > for run very well on the phi (we haven't tested these codes yet). The > codes we have tested are trivially parallel and very FP heavy - they ported > easily to the phi and run very well. > > The codes I tested (in like 2hrs) saw linear speedup to 60cores and then a > "kink" in performance and then continued performance gains right up to 240 > threads. Essentially these codes are single cpu with a trivial wrapper > around them to hand out work. This is exactly what hyper threading was > designed to help. So at 240 threads, we were about 120 times faster than a > single thread of this code. At 60 threads, we were 60 times faster :) > > Again, since the codes I tested were small data in, small data out and > heavy compute and trivially parallel, running over multiple phi's is > trivial and provide linear performance gains. As we start porting more of > our complex codes, I expect to see similar gains. Our codes already run > very very well on 64 cores… > > The phi's are separate cards, in separate pcie slots. I have not delved > into the programming api's fully, but I suspect you can utilise the one phi > card for your threaded codes. The way I've been running is with a native > phi application (basically using the Phi as a separate linux cluster node)… > using it in offload mode is very different and you may well be able to get > your kernel running across both with the right pragmas. > > To be 100% honest, we took the boots and all approach. If we only > purchased 1 phi to test on, we would never expend the energy to port all > our codes. Purchasing hundreds of them gives you a lot of impetus to port > your codes quickly :) > > > -- > Dr Stuart Midgley > sdm...@sdm900.com > > > > > On 13/02/2013, at 12:38 AM, Richard Walsh <rbwcn...@gmail.com> wrote: > > > > > Hey Stuart, > > > > Thanks for your answer ... > > > > That sounds compelling. May I ask a few more questions? > > > > So should I assume that this was a threaded SMP type application > > (OpenMP, pthreads) or it is MPI based? Is the supporting CPU of the > > multi-core Sandy Bridge vintage? Have you been able to compare > > the hyper-threaded, multi-core scaling on that Sandy Bridge side of the > > system with that on the Phi (fewer cores to compare of course). Using > the > > Intel compilers I assume ... how well do your kernels vectorize? Curious > > about the observed benefits of hyper-threading, which generally offers > > little to floating-point intensive HPC computations where functional unit > > collision is an issue. You said you have 2 Phis per node. Were you > > running a single job across both? Were the Phis in separate PCIE > > slots or on the same card (sorry I should know this, but I have just > > started looking at Phi). If they are on separate cards in separate > > slots can I assume that I am limited to MPI parallel implementations > > when using both. > > > > Maybe that is more than a few questions ... ;-) ... > > > > Regards, > >
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
