On Mon, Mar 15, 2010 at 07:30:15PM -0400, Patrick Geoffray wrote: > However, things are different for tiny packets. The minimum packet size > on Ethernet is 60 Bytes. The maximum packet rate (not coalesced !) is > 14.88 Mpps on a 10GE link, counting everything (inter-packet gap, CRC, > etc). If you do the math, that's 14.88*60 = 892 MB/s on the link, or 684 > MB/s if you remove the 14B Ethernet header (54% efficiency).
There's the additional complexity, for tiny packets, that different cards will have different outgoing inter-packet gaps, usually greater than the minimum. Switches can merge streams from multiple hosts and reduce that inter-packet gap on the receiving side, if multiple hosts talk to one. This is true for both Ethernet and IB. And if we're talking MPI, different MPIs have different header sizes. That's where a graph of non-coalesced MPI bandwidth (not including all the overheads) as a function of message size and core counts is interesting. The spreadsheet can kinda be useful for large messages, but not for data sizes < 1/2 MTU. But even for large data sizes, there are plenty of other factors which your real application can trip over. -- greg _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
