On Aug 9, 2008, at 5:03 PM, Reuti wrote:
Hi,
Am 09.08.2008 um 20:53 schrieb jitesh dundas:
We could try and implement this functionality of resuming broken
downloads like in some softwares like Download Accelerator and
bit-torrent.
I hope my views can help, so here goes:-
When a file is being downloaded, we can keep a stack of all of these
downloads in progress at a centralized repository, preferably where
the user has kept his file hosted for download or on the machine
where
the download is to be done.
Next, we can keep the track of the point at which the download
stopped
and store it in the repository. Next, if the user tries to start the
download it again, we can again retrieve it back from the data and
get
the end point of the previous download.
The end point for each file can include the file details in terms of
bits and bytes( 0 & 1) or even in percentages or pieces..Next time we
can break our file based on pieces or percentages( as needed) and
start the download from the nearest point that is best suited for the
user.
regarding user transmission of big files, maybe even between sites,
I would look into splitting the files and using a checksum like .par
or .par2.
http://sourceforge.net/projects/parchive
Even if one part doesn't make it to the other node, you can still
assemble the complete file due to the added checksum files.
But the original question was copying files inside a cluster to
thousands of nodes. As 1000 nodes still means some amount of money
to spend, what about looking into something like IBM's GPFS and
their SAN switch and connect all nodes to this switch?
-- Reuti
You may want to look at http://loci.cs.utk.edu. If you need to
distribute large files within a cluster or across the WAN, you can use
the LoRS tools to stripe the file over multiple servers and the
clients then try pulling blocks off of each server in parallel. Using
Internet2 and one client at Vanderbilt and a couple servers at Univ of
Tennessee, they were able to saturate UT's ~400 Mb/s I2 link (much to
the disbelief of the Vandy IT staff). I have seen ~5 Gb/s within a
cluster using good 10G NICs. :-)
Scott
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
