On Wed, 29 Oct 2014 12:02:49 -0400 Prentice Bisbal <prentice.bis...@rutgers.edu> wrote:
> > On 10/28/2014 06:24 AM, John Hearns wrote: > > Here is a very good post from Glenn Lockwood regarding FDR versus > > dual-rail QDR: > > > > > > http://glennklockwood.blogspot.co.uk/2013/05/fdr-infiniband-vs-dual-rail-qdr.html > > From the article: > > > The vendors would have you believe that FDR is just better in every > > way (after all, FDR's 14Gbit peak is bigger than QDR's 10Gbit > > peak), but I was skeptical. > > I thought that FDR was 54 Gb/s. I had to check Wikipedia. He's > talking about 1x (single-lane) performance. Most IB is 4x. This thread is getting thick with bad information and confusion. Let me try to clear some up: FDR is fourteen-rate (per lane) -> 14 * 4 lanes typical -> 56 Gbps on the wire. Encoding is 64/66 bit and after pci-express transfer size etc. you get around 6000 MB/s. QDR is 10G (per lane) -> 10 * 4 lanes typical -> 40 Gbps on the wire. Encoding is 8/10 -> 32 Gbps and after pci-express 3200-4000 MB/s. On a certain centos-6.5 platform I see these actual numbers using ib_write_bw/ib_write_lat (verbs): bw lat(near cpu) lat(far cpu) -------------------------------------------------- QDR 3300* 1.12 1.54 FDR 6000 0.85 1.59 QDR hw is mlx connectx3 to mlx switchX (same switch), ivy bridge node. FDR hw is mlx connectx3 to mlx switchX (same switch), sandy bridge node. Of course what people refer to as QDR could also be connectx2 or qlogic/intel. The smaller hit for "far cpu" for qdr could in my case be an improvment from snb to ivb. * I expected more like 4000 here but 3300 is what I got. /Peter _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
