On Mon, Dec 02, 2013 at 08:41:26AM -0500, atchley tds.net wrote: > On Mon, Dec 2, 2013 at 8:37 AM, atchley tds.net <atch...@tds.net> wrote:
> > I am not sure what Aries currently offers that IB does not. The IB in question is the True Scale adapter, which does some things really fast and other things pretty slowly. Aries has different features (quite different and more capable than IB, really), and is much larger. To put this into perspective, I suspect the typical modern Ethernet adapter has more gates than True Scale. If you're going to add something to a CPU, it had best be small. CPU guys get really irate if you reduce their yield. > > As Myricom showed with MX over Ethernet followed by > > Mellanox with RoCE, you can get low latency over Ethernet bypassing the > > kernel and the TCP stack. Indeed, Myricom+MX is quite similar in concept to the IB extension found in True Scale. The main difference (in my mind) is that OpenMX is hosted on a not-optimized-for-MX generic ethernet adapter, and that Myrinet's hardware was not fully optimized to do exactly what MX needed, nothing more and nothing less. True Scale is the smallest possible adapter that supports the basics of what MPI needs. The fabric is pretty irrelevant, as long as it has flexible routing. (See below for comments about SDN.) > > HPC sends a lot of small messages and various stacks are making use of > > 8-byte atomics. It is unhelpful to have a 64 byte minimum frame size in > > this case. Yes, a smaller frame size is quite nice for achieving high message rates for tiny packets. > > Ethernet topology discovery protocols were designed for environments where > > equipment can be changed out, expanded, or otherwise altered. This has changed in the new SDN (software defined networking) world. You can think of SDN on Ethernet as Infiniband management protocols implemented in ethernet, making many of the same mistakes that Infiniband did, plus some new ones. > Ethernet requires a single-path between any two endpoints. This is not true. It's more accurate to say that ethernet (especially TCP) benefits from in-order delivery, which you can ensure either on a host-host basis (which is what spanning tree provides) or on a per-flow basis (which is what SDN allows.) Personally, I'm a bit bummed this won't happen until 2015 :-( but I'm really excited to see True Scale's basic design continue into another generation. -- greg _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
