If anyone is interested in infrastructures using Ethernet like that, have a look at JASMIN int he UK
http://www.ceda.ac.uk/projects/jasmin/ http://www.channelweb.co.uk/crn-uk/news/2339003/viglen-scents-hpc-success-with-gbp4m-jasmin-deal Mellanox switching with 10Gbps to the edge and a rather large 40Gbps core infrastructure. Put some of the switches in with my own fair hands! ________________________________________ From: Beowulf [[email protected]] on behalf of Stu Midgley [[email protected]] Sent: 02 June 2016 05:01 To: Greg Lindahl Cc: Beowulf List Subject: Re: [Beowulf] curiosity killed the cat just need to workout how to build large fat-trees with ethernet. With 100G we get by with mlag for the spines (upto 640 hosts on 10g). with our 40G/10G network, we are using ospf/mpls/vpls to give a static fat-tree... 30k lines of config applied to ~20 switches. Nasty. On Thu, Jun 2, 2016 at 11:57 AM, Greg Lindahl <[email protected]<mailto:[email protected]>> wrote: On Thu, Jun 02, 2016 at 07:48:10AM +0800, Stu Midgley wrote: > That's it. As I said, I haven't used it since the early 00's. > > With 100Gb becoming common it might be time for these sort of MPI's to come > back. You could do better than these old interfaces, too... modern ethernet chips have multiple send/receive queues, so you could hack up an interface which looks a lot more like InfiniPath's one-queue-per-core model -- the OpenIB code for InfiniPath also has code that maps M cores:N queues if M>N. -- greg -- Dr Stuart Midgley [email protected]<mailto:[email protected]> Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Employees of XMA Ltd are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to company policy and outside the scope of the employment of the individual concerned. The company will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. XMA Limited is registered in England and Wales (registered no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
