I agree that all of the options (Infiniband, Myrinet, and 10 Gb
Ethernet) are too expensive. I have been looking into the low latency 10
Gb Ethernet cards from NetEffect, which use the iWARP specifications to
provide low latency. I haven't done any testing, yet, but the numbers
that they are releasing show them competitive with Infiniband/Myrinet as
the number of processes increase. Plus, I expect 10 Gb switches to
rapidly drop in price. I believe that the only economical solution in
the short term (3-5 year range) will be Ethernet based since "everyone"
knows Ethernet. It is only by selling substantial volume that the prices
drop to inexpensive. I foresee motherboard manufacturers placing 10 Gb
Ethernet adapters on-board server motherboards soon enough; I see no
reason why they can't be the low latency varieties.
I hope to start testing some of the NetEffect 10 Gb cards soon and will
try and post some numbers. Here is a link to some of the numbers that
NetEffect is publishing:
http://www.hpcwire.com/hpc/716435.html
Jess
Vincent Diepeveen wrote:
hi,
Thanks for your reaction.
Ethernet is of course too slow in latency.
the cheapest cable i see is 1 meter and $70
Cheapest card i see is $715
So the node price starts at $765, which is already way way more than
the total price of 1 node.
Now we didn't discuss the switches yet. Switches and routing of a
network is important.
The problem of myrinet nowadays is already that it is way too
expensive when compared to the node price.
I also tend to remember a few years ago that a myrinet card was like
far under $500.
Now cheapest card of myri i see is $715, and i didn't see the huge
price of switches
yet that will add up to node price.
More interesting than paying a $1000 a node for 10 gigabit MPI, is
having some older card say 3 gbit/s,
which uses MPI and is DMA low latency with a bit older switch for say
$300 a node.
Then you've got a good low latency network for a small price, yet
still making price of a node more expensive,
from $450 to $750.
Vincent
----- Original Message ----- From: "Jess Cannata" <[EMAIL PROTECTED]>
Cc: <beowulf@beowulf.org>
Sent: Friday, May 18, 2007 10:44 PM
Subject: Re: [Beowulf] Network considerations for new generation cheap
beowulfcluster
While I can't foresee the future, I do think that we are going to a
lot more low latency 10 Gb/s cards that use standard 10 Gb switches
and cables such as Myricom's 10 Gb Myrinet/Ethernet card and
NetEffect's 10 Gb Ethernet card.
http://www.myricom.com/Myri-10G/product_list.html
http://www.neteffect.com/ne020-features.html
Jess
Vincent Diepeveen wrote:
hi All of you,
Now that developments go fast in CPU land, by 22 july or so, intel
drops price of its quad core to $266 more or less.
Hopefully AMD's quadcore chip releases soon too for a decent price.
As intels memory subsystem is real weak, not to mention the extra
price that AMD and intel ask for dual socket/quad socket capable
chips, the optimal node is a single socket node.
4 cores is already a lot anyway for 1 highend network card.
That means in short that you can produce for quite little money,
far under $500, a node with 4 cores,
or considering the far higher taxrates in Europe, far under 500 euro
in Europe.
Basically what a node needs is a mainboard, a bit of RAM, and a cpu
with cooler. That keeps a node tiny and it's easier coolable. With
some wood then you can build a great case that holds many nodes.
Booting of course diskless over the gigabit network. Of course
interesting to know secondly is whether putting in ECC-reg ram is
interesting, considering its scandaleous high price always.
What are opinions here?
Of course now the question is how to get a reasonable low latency
highend network with a reasonable bandwidth (latency bigger priority
than bandwidth of course) and of course being capable of reading in
memory without writing. Of course the switch/routing prices + cable
prices need to be included in those price considerations.
Perhaps some bit older generation card gets sold very cheap now.
What are the options the coming years there, any manufacturer
keeping up with the dropped price of a single quad core node?
Gigabit ethernet is not an option of course, that just works for
embarrassingly parallel software, it's usually interrupting bigtime
the cpu and has an ugly one-way pingpong latency, especially when
there is several threads simultaneously shipping messages.
What are the options for the network in the future?
Vincent
------------------------------------------------------------------------
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
