Mark Hahn wrote:
application environments. Contrary to the detractors of the
technologies comments, the TOE/RDMA card *did* provide fairly
significant performance delta for real apps running MPI over gigabit
ethernet.
well, let's not conflate these things: TOE and RDMA are not intrinsically
related...
Agreed, but you had both on this card, and you couldn't turn one off ...
I won't talk on the business side of them. We did see 4x better
(wallclock) time on customers StarCD calculations being run over the
TOE/RDMA engine than over the pure gigabit path.
do you have a sense for whether this was due to RDMA giving you lower
latency
by bypassing the stack (either host or the nic-toe stack)? I don't think
I've ever heard of TOE itself providing any latency benefit.
Usually a detriment to latency, but as I remember, they set up a fast
path for TCP offload via an iWarp/RDMA engine. So it bypassed the stack.
also the more appropriate comparison is not a normal mpi-tcp-ip stack, but
rather something like GAMMA or OpenMX. I don't suppose you had a chance
to test those, did you?
Sadly no. We are playing with OpenMX now though, for a different code.
And Ammasso is long gone.
Customer was pleased until they realized that the design of this card
effectively killed their maximum memory size (3.1 GB vs the 4 GB
installed). This was the iWarp bit.
I'm guessing that techniques from the IB world would ameliorate this...
:)
I personally don't see the value in RDMA for gigabit (10 GbE yes), when
IB is ~$180/port (HCA + cable) on the client side and ~$50/port on the
switch side. The gigabit switch you need for the low latency bits are
around $100/port, and then the offload/RDMA NIC is > $500/port ....
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: [EMAIL PROTECTED]
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
