Mark,
I have been pleasantly surprised by the performance of Chelsio. My tests are very small scale (no switch) and the hardware I'm testing is for my Limulus desk side cluster. I ran netpipe TCP and got latencies of 9us back-to-back (SFP+) Hopping through a two port adapter I got 18 us. I then tried openMX. On omx_perf I got the following latencies, 0 hops 5.503 us (Head to n0) 1 hops 10.438 us (Head to n1) 2 hops 15.829 us (Head to n2) The network is a simple linear connection. looks like this head --- n0 --- n1 --- n2 I ran HPL and the results went from 395.2 GFLOPS with 1 GbE to 567.4 GFLOPS with 10 GbE. I saw similar improvements in NAS latency sensitie codes. The white paper with the data is being prepared right now. These test were for Chelsio T420-SO-CR (low cost stripped down 2 port NICS) I just received a 4-port 10 GbE card that I'm going to test with both the Chelsio T520-SO-CR (2 port) and the Mellanox MCX311A (1 port) The idea is 4-ports on the head node connected to each of three compute nodes, with the remaining port open for expansion. For my purposes, small low cost desk-side HPC, 10-GbE is an excellent option. -- Doug > Hi all, > I'd appreciate any comments about the state of 10G as a reasonable > cluster network. Have you done any recent work on 10G performance? > > https://lwn.net/Articles/629155/ > > shows some excellent evidence-based work on kernel paths, but it > seems focused on bulk/saturation, rather than single-packet latency. > (not that 10G MPI wouldn't be saturated in a lot of cases vs FDR!) > > I also read some comments recently about improvements that particularly > benefit certain Intel adapters - anyone have comments on Intel vs > Broadcom (vs Mellanox, I suppose). > > It seems like 10gT is on the cusp of real volume-type prices: I saw a > quote today for a major-vendor 24pt switch for something like $140/port > after educational discounts. I know 10gT is somewhat higher latency > than other PHYs - or is it worth sticking to SFP+? Any comments on > latency properties of various switches? Any nice results of recent > hardware and OpenMX? > > much appreciated! > -mark hahn > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -- > Mailscanner: Clean > -- Doug -- Mailscanner: Clean _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
