Dear all, I was wondering whether somebody has/had similar problems as I have.
We have recenctly purchased a bunch of new nodes. These are Sandybridge ones with Mellanox ConnectX-3 MT27500 InfiniBand connectors and this is where I got problems with. I am usually using Debian Squeeze for my clusters (kernel 2.6.32-5-amd64). Unfortunately, as it turned out I cannot use that kernel as my Intel NIC is not supported here. So I upgraded to 3.2.0-0.bpo.2-amd64 (backport kernel to sqeeze). Here I got network but the InfiniBand is not working. The device is not even recognized by ibstatus. Thus, I decided to do an upgrade (not dist- upgrade) to wheezy to get the newer OFED stack. Here I get the InfiniBand working but only with 8.5 Gb/sec. A simple reseating of the plug increases that to 20 Gb/sec (4X DDR), which is still slower than the speed of the older nodes (40 Gb/sec (4X QDR)). So I upgraded completely to wheezy (dist-upgrade now) but the problem does not vanish. I re-installed squeeze again and installed a vanilla kernel (3.8.8) and the latest OFED stack from their site. And guess what: same experiences here: After a reboot the IfniniBand speed is 8.5 and reseating the plug increases that to 20 Gb/sec. It does not matter whether I connect to the edge switch or to the main switch, in both cases I got the same experiences/observations. Frankly, I am out of ideas now. I don't think the observed speed change after reseating the plug should happen. I am in touch with the technical support here as well but I think we both are a bit confused. Now, am I right to assume that the Mellanox ConnectX-3 MT27500 are QDR cards so I should get 40 Gb/sec and not 20 Gb/sec? Has anybody made similar experiences? Any ideas? All the best from London Jörg -- ************************************************************* Jörg Saßmannshausen University College London Department of Chemistry Gordon Street London WC1H 0AJ email: j.sassmannshau...@ucl.ac.uk web: http://sassy.formativ.net Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
