Re: [Beowulf] Rsync - checksums

I didn't take it offensively, so no worries! Best, ellis On 6/18/19 1:52 PM, Bill Wichser wrote: Was it?  I meant it in the gracious of ways. We did not have that version of rsync and knew nothing of that call.  That pointer to an option was most appreciated. Bill On 6/18/19 11:46 AM, Mic

Re: [Beowulf] Rsync - checksums

Was it? I meant it in the gracious of ways. We did not have that version of rsync and knew nothing of that call. That pointer to an option was most appreciated. Bill On 6/18/19 11:46 AM, Michael Di Domenico wrote: On Tue, Jun 18, 2019 at 11:00 AM Bill Wichser wrote: Well thanks for THAT

Re: [Beowulf] Rsync - checksums

On Tue, Jun 18, 2019 at 11:00 AM Bill Wichser wrote: > Well thanks for THAT pointer! THAT was uncalled for. ___ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http

[Beowulf] Nvidia bring CUDA to ARM

By the end of the year Nvidia will have CUDA and the associated AI / HPC stuff available for ARM https://nvidianews.nvidia.com/news/nvidia-brings-cuda-to-arm-enabling-new-path-to-exascale-supercomputing ___ Beowulf mailing list, Beowulf@beowulf.org spons

Re: [Beowulf] Rsync - checksums

On 6/18/19 11:00 AM, Bill Wichser wrote: Well thanks for THAT pointer!  Using --checksum-choice=none results in speedup of somewhere between 2-3 times.  That's my validation of the checksum theory things have been pointing towards.  Now to get xxhash into rsync and I think we are all set. Gla

Re: [Beowulf] Rsync - checksums

if you get it working, I'd be interested in it :) I personally like the way tar does it, where you can provide your own "compression" which I've used to insert checksums into the stream. On Tue, Jun 18, 2019 at 11:03 PM Stu Midgley wrote: > Are you rsyncing over ssh? If so, get HPN-SSH and use

Re: [Beowulf] Rsync - checksums

No. Using the rsync daemon on the receiving end. Bill On 6/18/19 11:03 AM, Stu Midgley wrote: Are you rsyncing over ssh?  If so, get HPN-SSH and use the non-cipher. MUCH faster again :) On Tue, Jun 18, 2019 at 11:00 PM Bill Wichser > wrote: Well thanks for T

Re: [Beowulf] Rsync - checksums

Are you rsyncing over ssh? If so, get HPN-SSH and use the non-cipher. MUCH faster again :) On Tue, Jun 18, 2019 at 11:00 PM Bill Wichser wrote: > Well thanks for THAT pointer! Using --checksum-choice=none results in > speedup of somewhere between 2-3 times. That's my validation of the > check

Re: [Beowulf] Rsync - checksums

Well thanks for THAT pointer! Using --checksum-choice=none results in speedup of somewhere between 2-3 times. That's my validation of the checksum theory things have been pointing towards. Now to get xxhash into rsync and I think we are all set. Thanks, Bill On 6/18/19 9:57 AM, Ellis H. Wi

Re: [Beowulf] Rsync - checksums

On 6/18/19 9:16 AM, Bill Wichser wrote: Stock RH 7 version, rsync-3.1.2-6.el7_6.1.x86_64.  We've tried a number of recompiles.  gcc, Intel.  The only thing between identical compiles was the md4 vs md5. /bin/rsync -lptgoDAH -v --numeric-ids -d --relative --delete --delete-after --files-from=.

Re: [Beowulf] Rsync - checksums

Stock RH 7 version, rsync-3.1.2-6.el7_6.1.x86_64. We've tried a number of recompiles. gcc, Intel. The only thing between identical compiles was the md4 vs md5. /bin/rsync -lptgoDAH -v --numeric-ids -d --relative --delete --delete-after --files-from=... I'm not asking for help. Just if an

Re: [Beowulf] Rsync - checksums

On 6/18/19 6:59 AM, Bill Wichser wrote: Just for clarity here, we are NOT using the -c option.  The checksums happen whenever there is a transfer between the rsync source and the rsyncd on the other end. ...snip... This is not some trivial rsync running at the top level.  There is code we wrot

Re: [Beowulf] Rsync - checksums

we use rsync to a zfsonlinux fs with compression + dedup and snapshots for incrementals. works well :) means your most recent copy is a live fs people can access easily. On Mon, Jun 17, 2019 at 9:44 PM Bill Wichser wrote: > We have moved to a rsync disk backup system, from TSM tape, in order

Re: [Beowulf] Rsync - checksums

Just for clarity here, we are NOT using the -c option. The checksums happen whenever there is a transfer between the rsync source and the rsyncd on the other end. AFM does not work in our case. Our filesets are NOT independent and it would take months to make them so. We do not have enough

Re: [Beowulf] Rsync - checksums

On Mon, 17 Jun 2019 08:29:53 -0700 Christopher Samuel wrote: > On 6/17/19 6:43 AM, Bill Wichser wrote: > > > md5 checksums take a lot of compute time with huge files and even > > with millions of smaller ones.  The bulk of the time for running > > rsync is spent in computing the source and desti