>> that since we exist in 3d, our networks need to be a 3d lattice at scale. >> fighting power (flops, comm) seems like a noble, *engineering* fight, >> but fighting our existence's dimensionality is silly... > > We already exist in 3d - racks. We limit rack heights due to weight already > (another item to add to your list of limits above).
weight (and 2d footprint) are purely linear, and not very interesting, to me at least. it's demonstrably possible (not hard) to do racks in 3d - and in fact doing so generally draws more attention to the macro-structure of the cluster anyway (including airflow, for instance.) > I understand that you are arguing for 3d is silicon (or other substrate), > but there will still be a practical limit. I think I'm actually making the opposite argument. that at least for inter-unit networking, the topology for very scaled clusters needs to be a 2 or 3d mesh. if you're building a cluster laid out in 2d, as most are, then there are scaling problems with any non-2d topology. I love higher- order fabrics (fat trees, hypercubes, dragonfly, kautz, etc), but they have a sweet spot in their scale, and the whole point of exascale seems to be to exceed all sweet spots. within a "unit" (chassis, rack, take your pick), it's practical to exceed the "consensual dimensionality" - especially since you get to use different materials (say, a backplane with a complex pattern of connections implementing a 5d torus, etc). inside the package, the coming win seems to be simply 2.5d integration to minimize compute-memory distances, though this is nothing new to the IBM MCM world. > I saw some presentation a year or two ago that showed an exascale system as > a sphere with the switch complex in the center. It looked very 60s-ish, > proto-EPCOTish... works for me! though if you're going to do that, it would be awefully tempting to try to do free-space optics. but again, once you exceed the sweetspot for your free-space-optical-sphere-cluster, you have to pack multiple spheres, and that's going to be some kind of mesh... >> yes, I would like to demo your new blackhole-based interconnect! ;) > > Shouldn't it be wormhole-based? The blackhole-based one will just destroy > the data, no? I knew I should have just said "singularity" there! _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
