Hi Peter, what I am after in the end is installing the older DDR network which I got at quite newer machines (Ivy-bridges). The reason for that is budget, like so often these days, and hence I wanted to try it out first. Thus, I would have hoped that the bottle neck is the network and not the PCIe/PCI-X bus. I hope that makes sense.
All the best from London Jörg On Tuesday 28 Oct 2014 09:02:17 you wrote: > On Mon, 27 Oct 2014 22:38:26 +0000 > > Jörg Saßmannshausen <j.sassmannshau...@ucl.ac.uk> wrote: > > Dear all, > > > > thanks for the feedback. > > > > I think the best thing to do is to run a few test jobs and see what > > happens here. Once that is done I can report back so the community > > gets something back from me as well. > > Running tests is generally a good idea but remember that for almost all > combinations of infiniband generations and server/pcie generations one > port has been enough to eat all the pcie bandwidth. So using two ports > wont give more bandwidth. > > /Peter > > > I did not expect it to be that easy to be honest. I know that OpenMPI > > does test the various interfaces but I did not expect that it is > > using the two IB ports if they are available. > > > > So thanks for your help so far! > > > > All the best from London > > > > Jörg > > > > On Montag 27 Oktober 2014 Prentice Bisbal wrote: > > > On 10/27/2014 12:03 PM, Kilian Cavalotti wrote: > > > > Hi Jörg, > > > > > > > > On Mon, Oct 27, 2014 at 3:56 AM, Jörg Saßmannshausen > > > > > > > > <j.sassmannshau...@ucl.ac.uk> wrote: > > > >> I got some older dual Mellanox Technologies MT23108 cards (8.5 > > > >> Gb/sec (4X)) and currently I am only using one of the two ports > > > >> on them. I was wondering, is it possible with InfiniBand to > > > >> utilise the second port in such a way that I can increase the > > > >> bandwith of the network (and lower the latency maybe)? > > > > > > > > Last time I checked, the Linux bonding driver didn't allow to > > > > bond IB interfaces to increase throughput. I'd be happy to hear > > > > that this changed, but I think only redundancy modes (such as > > > > active-backup) were possible. > > > > > > > > Now, Open MPI can take advantage of multiple, independent ports > > > > connected to the same fabric: see > > > > http://www.open-mpi.org/faq/?category=openfabrics#ofa-port-wireup > > > > for details. > > > > > > > > Cheers, > > > > > > This makes sense. Since IB operates mostly in userspace, I would > > > expect setting up IB like this to done by the applications, not the > > > operating system. Since TCP/IP network is mostly handled by the > > > kernel, it would make sense that IPoIB would be configured by the > > > kernel, though. Feel free to correct me if this logic is wrong. > > > > > > You could increase the bandwidth, but not the latency. If anything, > > > the latency might go up as some additional work will need to be > > > done to coordinate the data going over the two different > > > connections. -- ************************************************************* Dr. Jörg Saßmannshausen, MRSC University College London Department of Chemistry Gordon Street London WC1H 0AJ email: j.sassmannshau...@ucl.ac.uk web: http://sassy.formativ.net Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
