That's correct
----- Original Message ----- From: beowulf-boun...@beowulf.org <beowulf-boun...@beowulf.org> To: Patrick Geoffray <patr...@myri.com> Cc: beowulf@beowulf.org <beowulf@beowulf.org> Sent: Mon Mar 15 16:45:44 2010 Subject: Re: [Beowulf] Q: IB message rate & large core counts (per node)? If I understand correctly, 40GbE is 64/66 encoded. Tom On 3/15/2010 5:30 PM, Patrick Geoffray wrote: > On 3/15/2010 5:24 PM, richard.wa...@comcast.net wrote: > >> to best and worst case). It would be good to add Ethernet to the mix >> (1Gb, 10Gb, and 40Gb) as well. >> > 10 Gb Ethernet uses 8b/10b with a signal rate of 12.5 Gb/s, for a raw > bandwidth of 10 Gb/s. I don't know how 1Gb is encoded and 40 Gb/s is > still in draft. Last time I looked at 40 Gb/s, it was pretty much four > 10 Gb links put together, so I would say 8b/10b with 50 Gb/s signal rate. > > >> >There is a similar protocol efficiency at the IB or Ethernet level, but >> >the MTU is large enough that it's much smaller compared to PCIe. >> >> Would you estimate less than 1%, 2%, 4% ... ?? >> > It depends on the packet size. For example, 14 Bytes Ethernet header on > 1500 Bytes MTU, that's 1%. For Jumbo frames at 9000B MTU, it's much less > than that. I don't know the header size in IB, but with an MTU of 2K or > 4K, it's negligible. > > However, things are different for tiny packets. The minimum packet size > on Ethernet is 60 Bytes. The maximum packet rate (not coalesced !) is > 14.88 Mpps on a 10GE link, counting everything (inter-packet gap, CRC, > etc). If you do the math, that's 14.88*60 = 892 MB/s on the link, or 684 > MB/s if you remove the 14B Ethernet header (54% efficiency). > > I don't think you can put all that on an Excel sheet :-) > > Patrick > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- -------------------------------------------------------------------- Tom Ammon Network Engineer Office: 801.587.0976 Mobile: 801.674.9273 Center for High Performance Computing University of Utah http://www.chpc.utah.edu _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
