> Date: Mon, 07 Nov 2011 13:16:00 -0500 > From: Prentice Bisbal<prent...@ias.edu> > Subject: Re: [Beowulf] building Infiniband 4x cluster questions > Cc: Beowulf Mailing List<beowulf@beowulf.org> > Message-ID:<4eb82060.3050...@ias.edu> > Content-Type: text/plain; charset=ISO-8859-1 > > Vincent, > > Don't forget that between SDR and QDR, there is DDR. If SDR is too > slow, and QDR is too expensive, DDR might be just right. And for DDR a key thing is, when latency matters, "ConnectX" DDR is much better than the earlier "Infinihost III" DDR cards. We have 100's of each and the ConnectX make a large impact for some codes. Although nearly antique now, we actually have plans for the ConnectX cards in yet another round of updated systems. This is the 3rd Generation system I have been able to re-use the cards in (Harperton, Nehalem, and now Single Socket Sandy Bridge), which makes me very happy. A great investment that will likely live until PCI-Gen3 slots are the norm. -- Da Bears?!
> -- > Goldilocks > > > On 11/07/2011 11:58 AM, Vincent Diepeveen wrote: >> > hi Prentice, >> > >> > I had noticed the diff between SDR up to QDR, >> > the SDR cards are affordable, the QDR isn't. >> > >> > The SDR's are all $50-$75 on ebay now. The QDR's i didn't find cheap >> > prices in that pricerange yet. >> > >> > If i would want to build a network that's low latency and had a budget >> > of $800 or so a node of course i would >> > build a dolphin SCI network, as that's probably the fastest latency >> > card sold for a $675 or so a piece. >> > >> > I do not really see a rival latency wise to Dolphin there. I bet most >> > manufacturers selling clusters don't use >> > it as they can make $100 more profit or so selling other networking >> > stuff, and universities usually swallow that. >> > >> > So price total dominates the network. As it seems now infiniband 4x is >> > not going to offer enough performance. >> > The one-way pingpong latencies over a switch that i see of it, are not >> > very convincing. I see remote writes to RAM >> > are like nearly 10 microseconds for 4x infiniband and that card is the >> > only one affordable. >> > >> > The old QM400's i have here are one-way pingpong 2.1 us or so, and >> > QM500-B's are plentyful on the net (of course big disadvantage: needs >> > pci-x), >> > which are a 1.3 us or so there and have SHMEM. Not seeing a cheap >> > switch for the QM500's though nor cables. >> > >> > You see price really dominates everything here. Small cheap nodes you >> > cannot build if the port price, thanks to expensive network card, >> > more than doubles. >> > >> > Power is not the real concern for now - if a factory already burns a >> > couple of hundreds of megawatts, a small cluster somewhere on the >> > attick eating >> > a few kilowatts is not really a problem:) >> > _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
