One thing you that im not seeing factored in is rpm speed of the drives. On 26 Jul 2014 15:05, "L. A. Walsh" <[email protected]> wrote:
> Adam Edgar wrote: > >> It seems the issue is indeed in the ssh layer. scp has the same issue and >> some work has been done in “fixing” that: >> >> http://www.psc.edu/index.php/hpn-ssh >> >> From the papers abstract: >> Status: O >> >> SCP and the underlying SSH2 protocol implementation in OpenSSH is >> network performance limited by statically defined internal flow control >> buffers. These buffers often end up acting as a bottleneck for network >> throughput of SCP, especially on long and high bandwith network links. >> >> > ---- > It is *A* bottle neck over networks. look for extensions to ssh to > ship unencrypted data streams. > There's a patch for this @ http://www.psc.edu/index.php/hpn-ssh. > > However, rsync is dog slow locally as well for exactly the reasons you > mention. > > An extract from another note on this topic (came up on suse list this > week). > > Someone suggested compression for a speed up... I responded to that: > > > On a local copy or local network, that usually slows down transfers. > > [**** 1000:1 speed ratio with large vs. small io sizes):] > > One might ask why rsync is so slow -- > copying 800G from 1 partition to another via xfsdump/restore takes a bit > under 2 hours, > or about 170MB/s, but with rsync, on the same partition with rsync > transfering > less than 1/1000th as much (700MB), it took ~70-80 minutes... or about > 163kB/s. > > That's on the same system (local drive -> another local drive) > > Transfer speeds depend on many factors. One of the largest is transfer > size (how much > transfered with 1 write /read. > Transfer 1GB, 1-meg at a time, took 2.08s read, and 1.56s to write (using > direct io). > > Transfer it at 4K: 37.28s, to read, and 43.02s to write. > > So 20-40x can be accounted for just on R/W size (1k buffers were 4x > slower). > > Many desktop apps still think 4k is a good "read size" > > Over a network, causes drop from 500MB/s down to less than 200KB/s > (as seen in FF and TB) -- 2500X. > > Optimal i/o size on my sys is between 16M-256M. > > So -- to answer your question, MANY things can affect speed, but I'd look > at the > R/W size first. > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/ > mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
