Hello, I have been hoping to find the time to see what it would take to use both the GAMMA and my FNN stuff together. Unfortunately, I've not been able to muster up that free time... As I alluded to in a previous post, I should have my FNN runtime support software available for public consumption sometime this summer... after my Ph.D. defense. Basically this FNN runtime support looks like an ethernet channel bond, but with the FNN driver knowing that depending on the destination of a packet, it can only use particular NIC(s) to get there. There is more to it than that, but hey, that's why I'm writing a dissertation on it plus some other FNN stuff. ;-)
Of course, the runtime support would have to be done differently for FNN+GAMMA, but I don't know the details on it right now. From what I remember looking through the GAMMA docs+source code a few years ago, it doubt it would be trivial. :-( As for a 72 node cluster, a Universal FNN with 48-port switches would need 3 switches, and two NICs per node. That FNN looks like a triangle, and is pretty easy to visualize the wiring pattern. Our handy FNN design CGI is here for those who haven't seen this before: http://aggregate.org/FNN/ BTW- Anyone have good or bad reports on GigE 48-port switch performance? A colleague has been having some mixed results with a D-Link DGS-1248T on an Athlon64 cluster (NForce4 chipset with forcedeth ethernet driver). Not enough trouble to warrant a full blown performance investigation, but sometimes things don't seem to go as fast as expected. For now they are saying "good enough" and just using the cluster. > > Does it still make sense to have a low-latency communication library for > > Gigabit Ethernet? Definitely! As others have posted, it would be cool if there was some form of GAMMA-lite that would work without needing a custom ethernet driver for each kind of NIC. The e1000 is very common, but you don't tend to find Intel NIC parts built into motherboards for AMD Athlon64s or Opterons. ;-) Well, time to get back to my cave to continue writing... On 2/4/06, Mark Hahn <[EMAIL PROTECTED]> wrote: > > GAMMA only supports the Intel PRO/1000 Gigabit Ethernet NIC (e1000 driver). > > well, that's the sticking point, isn't it? is there any way that GAMMA > could be converted to use the netpoll interface? for instance, look at > drivers/net/netconsole.c which is, admittedly, much less ambitious > than supporting MPI. > > > Latency at MPI level is below 12 usec, switch included (6.5 usec > > back-to-back). > > it would be interesting to know the latency of various GE switches - > I believe quite a number of them now brag 1-2 us latency. > > > Does it still make sense to have a low-latency communication library for > > Gigabit Ethernet? > > I certainly think so, since IMO not much has changed in some sectors of > the cluster-config space. AFAIKT, per-port prices for IB (incl cable+switch) > have not come down anywhere near GB, or even GB prices from ~3 years ago, > when it was still slightly early-adopter. > > my main question is: what's the right design? I've browsed the gamma > patches a couple times, and they seem very invasive and nic-specific. > is there really no way to avoid this? for instance, where does the > latency benefit come from - avoiding the softint and/or stack overhead, > or the use of a dedicated trap, or copy-avoidance? > > further, would Van Jacobson's "channels" concept help out here? > http://www.lemis.com/grog/Documentation/vj/lca06vj.pdf > > channels are a "get the kernel out of the way" approach, which I think > makes huge amounts of sense. in a way, InfiniPath (certainly the most > interesting thing to happen to clusters in years!) is a related effort, > since it specifically avoids the baroqueness of IB kernel drivers. > > the slides above basically provide a way for a user-level TCP library > to register hooks (presumably the usual <IP:port>) for near-zero-overhead > delivery of packets (and some kind of outgoing queue as well). the > results are quite profound - their test load consumed 77% CPU before, > and 14% after, as well as improving latency by ~40%. > > yes, it's true that if you spend the money, you can get much better > performance with less effort (for instance, quadrics is just about > the ultimate throw-money solution, with InfiniPath similar in performance > but much more cost-effective.) > > but gigabit is just so damn cheap! tossing two 48pt switches at 72 $1500 > dual-opt servers in a FNN config and bang, you've got something useful, > and you don't have to confine yourself to [EMAIL PROTECTED] of coupling. > > regards, mark hahn. > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Tim Mattox - [EMAIL PROTECTED] http://homepage.mac.com/tmattox/ I'm a bright... http://www.the-brights.net/ _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
