andrew holway wrote:
Gus,
It works(ish) and people are doing it but my research has shown that
it is not yet stable. I have been talking to various companies
offering lustre support. They have all told me that they can do it but
none have been able to offer a reference site.
We are running our filesystems and MPI traffic over the same IB network.
We are having no problems with this configuration. The system consists
of two trees (each with ~70% bisection bandwidth) connected via an top level
tree
to share IB communications between the filesystems and the compute nodes.
One side of the tree has ~350 woodcrest nodes, the other ~250 harpertown nodes.
We don't run jobs between the two systems, but both systems share the same
filesystems.
Just to complicate matters, we are supporting both Rapidscale and Lustre
(v1.6.5.1) on our nodes. The most obvious job contention we have seen
on the IB network is at the filesystem, not between the filesystem traffic
and the MPI traffic.
We had some issues with the subnet manager initially, but we have worked through
them. The latest version of Lustre has been quite stable in our environment.
As another posted already stated, I suspect that this configuration
would be of issue with codes that are very latency sensitive. Our
codes are more latency sensitive than bandwidth sensitive, and we haven't
seen any significant issues (and the configuration has been stable so far).
Craig
I bet you the chaps behind roadrunner aren't going to be publishing
any downtime figures.
as mentioned by mark, If you try and force lots of stuff down the
tubes you are going to break something. I guess its a _bit_ like
torrents on a naff home router, lots and lots of small torrent
connections filling up the nat table which cannot purge itself fast
enough which mean that larger downloads time out as they fall off the
bottom of the table.
Data on how hard IB switches have to work would be interesting. I have
a feeling that many people are taking their fabric to the very edge
and back again!
Perhaps someone can shed more light on the problems?
ta
Andy
On Tue, Aug 5, 2008 at 10:25 PM, Gus Correa <[EMAIL PROTECTED]> wrote:
Hello Beowulf fans
Is anybody using Infiniband to provide both
MPI connection and parallel file system services on a Beowulf cluster?
I thought to have a storage node that would
serve a parallel file system to the beowulf nodes over IB
(something like a NFS on steroids).
The same IB net would also work as the MPI interconnect.
Is this design possible?
On a small cluster, does it require two separate IB physical networks (cards
and switch),
or can it be done with a single IB card per node and one switch?
Is this design efficient?
Are there other practical and cost effective alternatives to this idea?
Would this type of design work with GigE instead of IB?
I confess I know nothing about parallel file systems and IB.
So, please forgive me if my questions are nonsense.
I also appreciate any links to readings that would mitigate my ignorance
on these subjects.
Thank you,
Gus Correa
--
---------------------------------------------------------------------
Gustavo J. Ponce Correa, PhD - Email: [EMAIL PROTECTED]
Lamont-Doherty Earth Observatory - Columbia University
P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
_______________________________________________
Beowulf mailing list, [email protected]
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, [email protected]
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
--
Craig Tierney ([EMAIL PROTECTED])
_______________________________________________
Beowulf mailing list, [email protected]
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
