Can you let rsync do the networking? If rsync isn't doing the networking then it isn't much more capable than cp yet it is significantly slower than cp.
On 12/18/2016 09:13 AM, Adam Nielsen wrote:
> Hi all,
>
> I'm wondering whether it is feasible to have an option that will make
> rsync spawn a separate thread to close files it has created, to avoid
> the main process blocking while the destination files are flushed during
> the close operation?
>
> The reason I ask is that it is currently very slow to use rsync on a
> fast locally-mounted network filesystem, as you see the following
> behaviour:
>
> 1. rsync reads the source file, network remains idle
> 2. Kernel buffers start to fill up
> 3. Some seconds later, kernel starts writing data to destination
> filesystem
> 4. rsync gets to the end of the file and closes the target file
> 5. rsync hangs for 10+ seconds while the target file's data gets
> flushed over the network. No data is being read from the source.
> 6. Back to step 1, rsync reads the next file from the source, while
> the network is idle as the kernel buffers are now empty.
>
> This gives the impression that the copy process alternates between
> reading then writing, instead of both reading and writing at the same
> time. It results in a much slower operation because in step 1 above,
> the network is idle, then in step 5, the local disk is idle.
>
> I am thinking that if the final close operation was performed in a
> separate thread, it would allow the main rsync operation to continue
> and start copying the next file while the previous one was still being
> flushed.
>
> This would mean both the source (local disk) and the target (network)
> would be fully utilized instead of them sitting idle for a large amount
> of the operation.
>
> Is something like this feasible?
>
> Many thanks,
> Adam.
>
> P.S. I am using a CIFS mount for this, and when I mount it with
> cache=none then the alternating read-then-write behaviour goes away,
> but the transfer rate drops by almost 30% so it ends up being slower
> overall.
>
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. [email protected] (work)
Orlando, Florida [email protected] (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
signature.asc
Description: OpenPGP digital signature
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
