On 01/23/2015 01:36 AM, Mark Hahn wrote:
Hi all,
I'd appreciate any comments about the state of 10G as a reasonable
cluster network. Have you done any recent work on 10G performance?
https://lwn.net/Articles/629155/
shows some excellent evidence-based work on kernel paths, but it seems
focused on bulk/saturation, rather than single-packet latency.
(not that 10G MPI wouldn't be saturated in a lot of cases vs FDR!)
I also read some comments recently about improvements that particularly
benefit certain Intel adapters - anyone have comments on Intel vs
Broadcom (vs Mellanox, I suppose).
It seems like 10gT is on the cusp of real volume-type prices: I saw a
quote today for a major-vendor 24pt switch for something like $140/port
after educational discounts. I know 10gT is somewhat higher latency
than other PHYs - or is it worth sticking to SFP+? Any comments on
latency properties of various switches? Any nice results of recent
hardware and OpenMX?
I've "this is the year of 10GbE" for something like the last 6 years,
and its not been generally true for HPC. We're at 40GbE now, and 100GbE
isn't far off (think months). 10GbE should be getting very cheap now,
but its not at copper RJ45 pricing. It is landed on motherboards, at
least a few.
We've had very good results with 10GbE from an aggregate bandwidth
scenario. Latency is hovering in the low double digits microseconds
(two way, 64 byte packets). One way latencies in the mid single digits.
If you have extremely latency sensitive stuff, Infiniband is better. If
you are mixed, 40 GbE is similar latency (low single digit microseconds,
and 3.2-ish GB/s) is much easier to configure (unless you do RoCE).
These days, you should choose between 40GbE and FDR IB (EDR is out real
soon now) for HPC.
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
e: land...@scalableinformatics.com
w: http://scalableinformatics.com
t: @scalableinfo
p: +1 734 786 8423 x121
c: +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
