On Wed, Aug 26, 2009 at 12:18:17AM +0200, Reuti wrote: > Am 25.08.2009 um 22:37 schrieb Nifty Tom Mitchell: > >> <snip> >> However, It does make sense to me to generate and maintain site >> specific meta >> data for all valuable data files to include both detection (yes tamper >> detection too) and recovery codes. > > Sounds like using local par or par2 files along with their hash > information about the original files. Maybe this could be implemented as > FUSE filesystem for easy handling (which will automatically split the > files, create the hashes and any number of par files you like). > > -- Reuti
Interesting... par2 looks close. It may solve my worries with Cloud storage. The differences between par and par2 are interesting. The various issues involving damaged files (even a single bit error) in the initial design of par were a limitation. http://www.par2.net/pardif.php I can see that there has been a lot of work done already making it is a good place to start. And yes slipping this under a FUSE filesystem might hide the pain for local storage. >> I would extend this to all data with >> the hope that any problems might be seen first on inconsequential >> files. >> Tripwire might be a good model for starting out on this. >> >> I should note that the three 'big' error rate problems I have worked on >> in the past 25 years had their root cause in an issue not understood >> or considered at design time so empirical data from the customer was >> critical. Data sheets and design document conclusions just missed the >> issue. >> These experiences taught me to be cautious with storage statistics. >> >> Looming in the dark clouds is a need for owning your own data >> integrity. >> It seems obvious to me in the growing business of cloud computing and >> cloud storage >> that you need to "trust but verify" the integrity of your data. My >> thought >> on this is that external integrity methods are critical in the future. >> >> And do remember that "parity is for farmers." >> >> >> >> -- >> T o m M i t c h e l l >> Found me a new hat, now what? >> a >> _______________________________________________ >> Beowulf mailing list, [email protected] sponsored by Penguin >> Computing >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > -- T o m M i t c h e l l Found me a new hat, now what? _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
