Geoff Galitz wrote:
Hi folks,
During an HPC talk some years ago, I recall someone mentioned a tool
which can copy large datasets across a cluster using a ring topology.
Perhaps someone here knows of this tool?
There are a few, commercial, and open source.
On the commercial side is exludus, xcp, and maybe one or two others.
Exludus is basically a file pre-caching mechanism. Java based. xcp (by
Scalable) is MPI based. It does a pretty good job of moving data. \
On the open source side, I havent seen things other than the udp
broadcast based tools (we had written one several years ago, named mcp),
but anyone using a cluster will tell you that udp broadcast can be very
detrimental to non-udp broadcast usage of the switch, say for logins,
NFS, command and control, ...)
More to the point, we are pushing around datasets that are about
1Gbyte. The datasets are pushed out to dozens of nodes all at once and
we foresee saturating the I/O system on our cluster as we grow. We are
limited to using just the available disks and are looking for a
reasonable solution that can support this kind of simultaneous access.
xcp might help.
Currently we push the data out using rsync, but if I don't get any
better ideas I may simply move to a pull system where the data is
fetched by HTTP. I can get better throttling that way, at least.
For a few dozen nodes, this might work.
Joe
-geoff
Geoff Galitz
[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
------------------------------------------------------------------------
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: [EMAIL PROTECTED]
web : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 734 786 8452 or +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
