a) the real big difference between highend and lowend nic's IMHO is
having DMA capabilities and the one way pingpong latency it gives.
Several persons replied and not a SINGLE ONE of them talks about
one way pingpong latency, which is one of the most important features
of a highend network for *many* applications.
It basically tells you in a primitive way: "my application at processor X
needs some data
from some far away node in order to continue its calculation, what time do i
need to
wait until i have that data?"
b) you quote your nic's there. Please give me the one way poingpong latency
from your NICs and also the PRICE of those nics. Not even asking after the
price
of the fiber network, as the billions of dollars that lie in the underground
is hard to
estimate anyway.
It's of course quite weird IMHO to pay 800 dollar for a 1 gigabit NIC,
if for a similar price you can pay some highend network card that has 10
times more
bandwidth and a 50 times better one way pingpong latency.
If your argument is: "but it works for the software in question which works
over tcp/ip and that
highend network card doesn't have a tcp/ip implementation", then that's a
very weird
way of escaping as it doesn't change the outcome that there is a huge
objective
difference between the highend network cards and the average $30 gigabit
ethernet card.
Note that when i try to transfer i/o over default gigabit ethernet between
some servers, using a
5 meters cross utp cable, that it's quite difficult to come above 20MB/s,
this where both raid10
arrays handsdown handle speeds of a 100 MB/s (to say it polite). If you want
to know the NIC
used. Just lookup which NICs the Tyan S2881 has.
This where further both servers were completely idle.
On the other hand at simplistic pci 64 bits 66Mhz (not exactly highend and
quite dated) the
transfer between 2 dual k7's was 330MB/s with 2 quadrics cards (admittedly
that wasn't
stored to disk but using the shmem).
Vincent
----- Original Message -----
From: "J. Andrew Rogers" <[EMAIL PROTECTED]>
To: "Vincent Diepeveen" <[EMAIL PROTECTED]>
Cc: <beowulf@beowulf.org>
Sent: Saturday, June 17, 2006 12:27 AM
Subject: Re: [Beowulf] Slection from processor choices; Requesting Giudence
On Jun 16, 2006, at 3:00 PM, Vincent Diepeveen wrote:
Jeff, we know how some people can mess up installs, but if you have
gigabit ethernet,
with a one way pingpong latency of like 50-100 us if you're lucky, which
is not using DMA i
guess by default; in short nonstop interrupting your cpu, versus newer
myri cards that are probably under 3 us, factor 20 to 30 advantage to
myri then you need to really not manage to install the network to let
gigabit ethernet work better.
I get a round-trip latency between Linux servers (vanilla CentOS install)
across a metro GigE IP network of ~140 us, which I guess means we are
"lucky". I would hope that a dedicated, tuned, and very local GigE
network could do a lot better than what I get between Internet servers
separated by kilometers of fiber and various bits of network gear. And I
know it can.
You exaggerate the limitations of GigE. Current incarnations of GigE can
be close enough to Myrinet that it is perfectly functional for some
applications and pretty competitive for someone willing to extract the
capability from it.
J. Andrew Rogers (who admittedly has an above average IP network)
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
