No. Using the rsync daemon on the receiving end.
Bill
On 6/18/19 11:03 AM, Stu Midgley wrote:
Are you rsyncing over ssh? If so, get HPN-SSH and use the non-cipher.
MUCH faster again :)
On Tue, Jun 18, 2019 at 11:00 PM Bill Wichser <b...@princeton.edu
<mailto:b...@princeton.edu>> wrote:
Well thanks for THAT pointer! Using --checksum-choice=none results in
speedup of somewhere between 2-3 times. That's my validation of the
checksum theory things have been pointing towards. Now to get xxhash
into rsync and I think we are all set.
Thanks,
Bill
On 6/18/19 9:57 AM, Ellis H. Wilson III wrote:
> On 6/18/19 9:16 AM, Bill Wichser wrote:
>> Stock RH 7 version, rsync-3.1.2-6.el7_6.1.x86_64. We've tried a
>> number of recompiles. gcc, Intel. The only thing between
identical
>> compiles was the md4 vs md5.
>>
>> /bin/rsync -lptgoDAH -v --numeric-ids -d --relative --delete
>> --delete-after --files-from=...
>>
>> I'm not asking for help. Just if anyone had attempted to change
the
>> algorithm into something much faster.
>>
>> I refer you to this project https://cyan4973.github.io/xxHash/
where
>> there is a table of speeds. Regardless of what anyone might
>> speculate, we are pursuing this route of changing out the
algorithm.
>> Maybe it's all for naught. Maybe it isn't. But in a few weeks
>> hopefully we'll have determined.
>
> Very interesting. From the rsync man page:
>
> "Note that rsync always verifies that each transferred file was
> correctly reconstructed on the receiving side by checking a
> whole-file checksum that is generated as the file is transferred,
but
> that automatic after-the-transfer verification has nothing to do
with
> this option’s before-the-transfer "Does this file need to be
updated?"
> check."
>
> So it sounds like you have sufficient churn in large files that the
> checksum validation post-transfer is your bottleneck. Short of
hacking
> rsync to use a faster algorithm, your remaining choice is to use the
> --checksum-choice=STR and set it to none, and then perform your own
> hashing out-of-band to check the transferred data using the list you
> have provided via in files-from. This will nerf rsync's ability
to do
> delta-transfer, which may be ok depending on the nature of your
churning
> files. If your pipes are huge (atypical for DR), your CPU is
weak, and
> your churning data is mostly completely new or completely changed
files,
> --checksum-choice=none may work very well for you.
>
> Best,
>
> ellis
>
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
<mailto:Beowulf@beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
--
Dr Stuart Midgley
sdm...@gmail.com <mailto:sdm...@gmail.com>
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
