Use a real operating system.
On 28 April 2013 09:36, Jörg Saßmannshausen <j.sassmannshau...@ucl.ac.uk> wrote: > Hi Josh, > > interesting. However, I am not using XEN on that machine at all and I don't > have the XEN kernel installed. Thus, that is not the problem. > > All the best from a sunny London > > Jörg > > > On Sonntag 28 April 2013 Josh Catana wrote: >> I noticed on systems running xen-kernel netback driver for virtualization, >> bandwidth drops to very low rates. >> >> On Apr 27, 2013 6:19 PM, "Brice Goglin" <brice.gog...@gmail.com> wrote: >> > Hello, >> > >> > These cards are QDR and even FDR, you should get 56Gbit/s (we see about >> > 50Gbit/s in benchmarks iirc). That what I get on sandy-bridge servers >> > with the exact same IB card model. >> > >> > $ ibv_devinfo -v >> > [...] >> > >> > active_width: 4X (2) >> > active_speed: 14.0 Gbps (16) >> > >> > These nodes have been running Debian testing/wheezy (default kernel and >> > IB packages) for 9 months without problems. >> > >> > I had to fix the cables to get 56Gbit/s link state. Without Mellanox FDR >> > cables, I was only getting 40. So maybe check your cables. And if you're >> > not 100% sure about your switch, try connecting the nodes back-to-back. >> > >> > You can try upgrading the IB card firmware too. Mine is 2.10.700 (likely >> > not uptodate anymore, but at least this one works fine). >> > >> > Where does your "8.5Gbit/s" come from? IB status or benchmarks? If >> > benchmarks, it could be related to the PCIe link speed. Upgrading the >> > BIOS and IB firmware help me too (some reboot gave PCIe Gen1 instead of >> > Gen3). Here's what you should see in lspci if you get PCIe Gen3 8x as >> > expected: >> > >> > $ sudo lspci -d 15b3: -vv >> > [...] >> > >> > LnkSta: Speed 8GT/s, Width x8 >> > >> > Brice >> > >> > Le 27/04/2013 22:05, Jörg Saßmannshausen a écrit : >> > > Dear all, >> > > >> > > I was wondering whether somebody has/had similar problems as I have. >> > > >> > > We have recenctly purchased a bunch of new nodes. These are Sandybridge >> > >> > ones >> > >> > > with Mellanox ConnectX-3 MT27500 InfiniBand connectors and this is >> > > where >> > >> > I got >> > >> > > problems with. >> > > >> > > I am usually using Debian Squeeze for my clusters (kernel >> > >> > 2.6.32-5-amd64). >> > >> > > Unfortunately, as it turned out I cannot use that kernel as my Intel >> > > NIC >> > >> > is >> > >> > > not supported here. So I upgraded to 3.2.0-0.bpo.2-amd64 (backport >> > >> > kernel to >> > >> > > sqeeze). Here I got network but the InfiniBand is not working. The >> > >> > device is >> > >> > > not even recognized by ibstatus. Thus, I decided to do an upgrade (not >> > >> > dist- >> > >> > > upgrade) to wheezy to get the newer OFED stack. >> > > >> > > Here I get the InfiniBand working but only with 8.5 Gb/sec. A simple >> > >> > reseating >> > >> > > of the plug increases that to 20 Gb/sec (4X DDR), which is still slower >> > >> > than >> > >> > > the speed of the older nodes (40 Gb/sec (4X QDR)). >> > > >> > > So I upgraded completely to wheezy (dist-upgrade now) but the problem >> > >> > does not >> > >> > > vanish. >> > > I re-installed squeeze again and installed a vanilla kernel (3.8.8) and >> > >> > the >> > >> > > latest OFED stack from their site. And guess what: same experiences >> > > here: After a reboot the IfniniBand speed is 8.5 and reseating the >> > > plug >> > >> > increases >> > >> > > that to 20 Gb/sec. It does not matter whether I connect to the edge >> > >> > switch or >> > >> > > to the main switch, in both cases I got the same >> > >> > experiences/observations. >> > >> > > Frankly, I am out of ideas now. I don't think the observed speed change >> > >> > after >> > >> > > reseating the plug should happen. I am in touch with the technical >> > >> > support >> > >> > > here as well but I think we both are a bit confused. >> > > >> > > Now, am I right to assume that the Mellanox ConnectX-3 MT27500 are QDR >> > >> > cards >> > >> > > so I should get 40 Gb/sec and not 20 Gb/sec? >> > > >> > > Has anybody made similar experiences? Any ideas? >> > > >> > > All the best from London >> > > >> > > Jörg >> > >> > _______________________________________________ >> > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing >> > To change your subscription (digest mode or unsubscribe) visit >> > http://www.beowulf.org/mailman/listinfo/beowulf > > > -- > ************************************************************* > Jörg Saßmannshausen > University College London > Department of Chemistry > Gordon Street > London > WC1H 0AJ > > email: j.sassmannshau...@ucl.ac.uk > web: http://sassy.formativ.net > > Please avoid sending me Word or PowerPoint attachments. > See http://www.gnu.org/philosophy/no-word-attachments.html > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
