On 6/17/19 1:35 PM, pellman.j...@gmail.com wrote:
I know that at one point, some Intel chips had instruction extensions available to speed up SHA checksums by computing them directly in hardware. Might be worth looking into: https://software.intel.com/en-us/articles/intel-sha-extensions
On my ancient desktop with a circa-2012 AMD processor md5sum executes on a cached 10GB file at roughly 650MB/s. I would expect a reasonably recent CPU to crush this. I expect one could max out a 25 or 40GbE link from a single client and a sufficiently fast CPU for m5sum on a modern processor.
I think we need more info from the OP describing why he thinks md5 is at fault before it makes sense to go down this rabbit hole.
But I agree with the general consensus that DR, especially for a 10PB system, is not best delivered via rsync alone.
Best, ellis _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
