> > I'd like to paralyze that across multiple nodes to drive the aggregate > up > > I was hoping someone would pop up say, hey use this magical piece of > software. (of which im unable to locate).. > My recommendation also would be to use an external storage device - a USB drive would be useful, and I have been involved in a couple of industrial projects where data has been brought to a cluster on an external USB drive. It is as people say quite an efficient way to transfer the data.
I gather that for high def digital cinema a RAID array is physically shipped to the cinema - I guess that also helps with data security, as you could do some sort of encryption on the drives, though I might be wrong. In the digital media world, there are some fast parallel SCP boxes which are an industry standard - I gather they cost $$$$ but do make transfers faster. I forget the name, and if they don't really do parallel SCP forgive me - its something along those lines. Re. moving data to/from a cluster over a WAN link, I did look at this recently. You can set up a fuse filesystem running over SSH. This actually works quite well from the point of view of ease of setting up and usability, but I didn't try any serious data transfer over it - and of course it cannot be faster than ssh anyway! I did also have a look at the types of tools used by grids for bulk data transfer, but not much more than looking. Here's an interesting link I found: http://fasterdata.es.net/tools.html ps. you don't say how you are transferring the data - if via rsync you have looked at the encryption options you are using? John Hearns The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy. _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
