Mark Hahn wrote: >> Doesn't this assume worst case all-to-all type communication >> patterns. > >I'm assuming random point-to-point communication, actually.
A sub-case of all-to-all (possibly all-to-all). So you are assuming random point-to-point is a common pattern in HPC ... mmm ... I would call it a worse case pattern, something more typical of graph searching codes like they run at the NSA. Sure a high radix switch (or better yet a global memory address space, Cray X1E) is good and designed for this worst-case, but not sure this is the common case data reference pattern in HPC ... if it were they would be selling more global memory systems at Cray and SGI (not just to the NSA). There you might also want a machine like the Cray XMT where the memory is flat and stalled threads can be switched out for another thread. >> If you are just trading ghost cell data with your neighbors >> and you have placed your job smartly on the torus the fan out >> advantage mentioned is irrelevant. No? > >if your comms are nearest-neighbor, then yes, a nearest-neighbor >fabric is your friend ;) I think that if you look at the HPC space globally there is still a lot of locality that you can rely on. Familiar with the "7 dwarves" paper from Berkeley? >how often does that actually happen? to work out so neatly would >preclude, for instance, adaptive meshes, right? it seems like mostly >I see jobs with no obvious regular structure to their communication. Really ... must be doing a lot of turbulent flow simulations with shedding vortices, crash simulations with self-penetrating meshes ... tough stuff for your average cluster or even your above average cluster. Even AMR codes usually attempt to discover new neighbors and localize them. Not disrespecting switches, but they are in a sense designed for worse case scenarios (the design asserts that "there are no neighborhoods") ... a torus design appeals to the middle ground were locality is not banished. rbw
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
