+1 for the 24 port flextronics switches. They are very cost effective for half bisectional networks upto 32 ports. It starts to get messy after that.
I wonder how long we will be waiting for switches based on the 36p asic? On Thu, Jun 12, 2008 at 4:08 PM, Don Holmgren <[EMAIL PROTECTED]> wrote: > > Ramiro - > > You might want to also consider buying just a single 24-port switch for your > 22 nodes, and then when you expand either replace with a larger switch, or > build a distributed switch fabric with a number of leaf switches connecting > into a central spine switch (or switches). By the time you expand to the > larger cluster, switches based on the announced 36-port Mellanox crossbar > silicon will be available and perhaps per port prices will have dropped > sufficiently to justify the purchase delay and the disruption at the time of > expansion. > > If your applications can tolerate some oversubscription (less than a 1:1 > ratio of leaf-to-spine uplinks to leaf-to-node connections), a distributed > switch fabric (leaf and spine) has the advantage of shorter (and cheaper) > cables between the leaf switches and your nodes, and relatively fewer longer > cables from the leaves back to the spine, compared with a single central > switch. > > We have many Flextronics switches - SDR and DDR, 24-port and 144-port - on a > pair of large clusters (520 nodes, and 600 nodes) built in 2005 and 2006. No > complaints. But, we have been self-supporting, and I would guess you would > have very different support structures with Voltaire or Qlogic. With the > Flextronics > switches you will definitely be using the OFED stack, and you will have to > run > a subnet manager on one of your nodes (dedicated is probably best). You > could > optionally buy an embedded subnet manager on the Voltaire or Qlogic > switches, > depending upon model, though I believe for a large fabric an external subnet > manager is still recommended. > > Don Holmgren > Fermilab > > > > > On Tue, 10 Jun 2008, Ramiro Alba Queipo wrote: > >> Hello everybody: >> >> We are about to build an HPC cluster with infiniband network starting >> from 22 dual socket nodes with AMD QUAD core processors and in a year or >> so we will be having about 120 nodes. We will be using infiniband both >> for calculation as for storage. >> The question is that we need a modular solution and we are having 3 >> candidates: >> >> a) Voltaire Grid Director SDR or DDR 288 ports (9988 or 2012 models)-> >> seems very good and well supported, but very expensive. >> >> b) Qlogic SilverStorm 9120 (144 ports) -> no price and support >> information yet >> >> c) Flextronics 10U 144 Port Modular-> very good at price but little >> support => risky option?. >> >> I am in a mess. What is your opinion about this matter? Are you using >> any of this products. >> >> Regards > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
