Hi, Thank you for taking the time to look at this..
On Mon, March 28, 2016 10:41 am, Dominic Raferd wrote: > Is this really your first rdiff-backup to this location? If you have any > previous rdiff-backup runs to this repository then the situation is > complicated by rdiff-backup's need to create a new set of reverse diff > files to be able to regress to previous file contents. Yes, this s really the first rdiff-backup to this location. A second backup run shortly after the first one completed finished in 55 minutes. > What is your /tmp location? rdiff-backup uses this location for some > operations though not AFAIK for standard backup runs. Still, if /tmp is on > encfs maybe it could be a culprit; you can override rdiff-backup's > temporary file location with --tempdir and --remote-tempdir. If it is truly /tmp then no; /tmp is a ramdisk on the backup server and is on the root disk on the target server. Neither are being run through encfs. If, however, you mean the rdiff-backup-data.tmp files, those ARE being run through encfs. > Might also be worth trying --ssh-no-compression. I already have "Compression no" set in ~/.ssh/config so I'm not sure what this would add? > Dominic > http://www.timedicer.co.uk -derek PS: I'm running a raw rsync command now just to see how it behaves -- so far I'm only seeing about 2MB/s, but it's only been running for 10 minutes or so. > > On 28 March 2016 at 14:37, Derek Atkins <[email protected]> wrote: > >> Just a quick update: I tried making these changes on both sides and it >> really didn't make a difference. Full backup of 202852072 Kbytes >> required 2267m25.913s (previously it took 2346m57.800s, so it only sped >> up a factor of 3%. >> >> Only thing I have not yet tried is running a raw rsync to see how fast >> that runs. I'll do that next. >> >> So, back to my orignal question: anyone have any idea how to get initial >> transfers to run faster (or indeed any significant data transfers)? >> >> Thanks, >> >> -derek >> >> "Derek Atkins" <[email protected]> writes: >> >> > Hi, >> > >> > I'm trying to use rdiff-backup to backup a bunch of servers. One >> > particular server contains about 160GB of data, but when I try to >> perform >> > the rdiff-backup it's saving the data at a measly 1MB/s. >> > >> > Here's my configuration: >> > >> > [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas] >> > >> > I ran a bunch of tests to try to figure out my bottlenecks. >> > >> > I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000) on >> the >> > backup server. Going directly to FreeNAS via NFS (bybassing encfs) I >> get >> > 50.2MB/s. If I run dd directly on the backup server (through encfs) I >> get >> > 20.1MB/s. If I go over SSH from the backup server to the target >> server >> > and run the dd on the target server, then write to FreeNas through >> encfs >> > declines to 7.6MB/s. >> > >> > Note that in those SSH tests, however, I forgot to turn off >> compression. >> > When I do that, the throughput for the dd test reduced to 6.6BM/s. >> (Note >> > that this is running simultaneously with a running rdiff-backup, so >> it's >> > possible that they are reducing performance). >> > >> > Then I ran an scp test to the same target server; copying about 1.4GB >> of >> > photos. Files ranged in size from 10KB to 5MB. When run in standard >> mode >> > (displaying each file status) I got 4.4MB/s. Running in quiet mode I >> get >> > 5.1MB/s. >> > >> > So clearly the bottleneck is in rdiff-backup -- performance (IMHO) >> should >> > not be significantly slower than the last dd-over-ssh test. It >> appears >> > rdiff-backup is slowing me down by a factor of 5x throughput versus >> scp. >> > >> > I found a message from Ben from 2005 where he suggests increasing the >> > blocksize and conn_bufsiz settings in Globals.py: >> > >> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html >> > >> > What he didn't say was whether this needed to be changed on the target >> > server, the backup server, or both. Nor do I know if that would >> actually >> > help this situation. >> > >> > Do you have any ideas? >> > >> > Thanks, >> > >> > -derek >> > >> > PS: According to rpm, both systems are running version 1.2.8. >> >> -- >> Derek Atkins 617-623-3745 >> [email protected] www.ihtfp.com >> Computer and Internet Security Consultant >> >> _______________________________________________ >> rdiff-backup-users mailing list at [email protected] >> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users >> Wiki URL: >> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki >> > -- Derek Atkins 617-623-3745 [email protected] www.ihtfp.com Computer and Internet Security Consultant _______________________________________________ rdiff-backup-users mailing list at [email protected] https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
