On Thu, 2008-07-17 at 04:42 +1000, Andrew Robbie (GMail) wrote: > On Tue, Jul 15, 2008 at 12:35 AM, Josip Loncaric <[EMAIL PROTECTED]> wrote: > > > > Another good link: > > > > http://www.lanl.gov/orgs/hpc/roadrunner/rrtechnicalseminars2008.shtml > > As I was reading the slides, one question leap out at me: they have a > huge IB network connecting every 'node', but instead of connecting > direct to storage, this connects to an IB-to-10GigE bridge/IO > processor board. Why? If they used avoided the protocol conversion > going on that would be inherently simpler, and I've seen nothing to > indicate that 10GigE is faster or cheaper (certainly not cheaper!). > > Is this (ethernet) a Panasas requirement? Yes indeed it is. I'll stick my head up above the parapet as someone who cares for and feeds several Panasas installations, though I don't work for the company.
For installations using Infiniband, Panasas will advise on how to implement Infiniband to (10gig) Ethernet storage routers. I take it here that Roadrunner is using a blade in the Infiniband switch, rather than a discreet router. This sounds a very good idea to me, and I looked into it for a particular project in the UK, though we didn't go for that approach in the end. To answer your question more directly, Panasas is a storage cluster to complement your compute cluster. Each storage blade is connected into a shelf (chassis) with an internal ethernet network. Each shelf is then connected to your ethernet switch with at least 4Gbps of bandwidth. It might look on the front like a big RAID array - and hence the questions as to why you don't have a fibrechannel or native Infiniband connector on it. But its not really a RAID array - its a storage cluster. Files are RAIDed over the filesystem and your client stripes IO over several storage blades at any one time. And please lets not start any Infiniband good/ethernet bad wars here, or bandwidth willy-waving. Panasas have made some damned good engineering decisions and their system scales like crazy - just what you need for something like Roadrunner. _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
