On 2018-09-14 04:37, G.W. Haywood via BackupPC-users wrote:
Hi there,
About a month ago I made the elementary mistake of upgrading my backup
server from Debian Jessie to Debian Stretch. Just about everything
broke.
The biggest concern was the backups themselves. The upgrade changed
BackupPC from version 3.3.0, which seemed to have been working fine for
years, to version 3.3.1, which didn't work at all. Wouldn't even
start.
Naturally, it would be incorrect to ascribe this breakage to BackupPC
3.3.1, which works well. Debian made a number of changes which would
directly affect BackupPC, all of which require adjustments.
(Alternatively, of course, you can start over.)
After a couple of days trying to get it to start I decided the best
thing
to do would be to cut my losses and install a version 4. So I
installed
version 4.2.1.
A couple of days later I had what I thought was a working 4.2.1 and
things
started to settle down, but 4.2.1 seemed to be eating a LOT more disc
space
than V3.x ever did and I spent another couple of days moving everything
I
could off the 3TB drive which holds the pool partition. V3 used just
over
1TB to back up 12TB of data on our various machines. V4 wanted more
like
2.6TB and it was still rising slowly four days after the initial
backups
had completed. Well they'd stopped, if not exactly completed - there
were
some 'partial' backups.
Still fearing "no space left on device" I used BackupPC_ls (it's
lovely!)
to look around in the backups, and then took my courage in both hands
and
tried used BackupPC_backupDelete to delete some cruft from home
directories
which really doesn't need to be backed up; entire directories like
'.cache'
and '.moonchild productions' from several home directories on several
hosts.
And I used "rm -r *" to list the contents of a directory.
In all seriousness, BackupPC_backupDelete is not a tool which removes
directories you did not mean to back up. It deletes entire backups or
shares.
That's when I think things really started to unravel. AFAICT when I
gave a
command to BackupPC_backupDelete to delete a directory from a backup,
it in
fact deleted the entire backup. More than once. Here's some debug
output
for an attempt to delete data from a backup for a remote host. I've
made a
couple of edits to shorten the bash prompt, split the long command line
with
a backslash escaped newline, and redact the user's login name:
8<----------------------------------------------------------------------
tornado:[...]/kestrel# >>>
/usr/share/backuppc/bin/BackupPC_backupDelete \
-h kestrel -n 271 -l -s Homes /xxxxxx/.cache '/xxxxxx/.moonchild
productions'
__bpc_pidStart__ 20851
BackupPC_backupDelete: removing #271/Homes
__bpc_progress_state__ delete #271/Homes
BackupPC_backupDelete: No prior backup for merge
__bpc_progress_fileCnt__ 17116
__bpc_progress_fileCnt__ 17918
__bpc_progress_fileCnt__ 28734
bpc_attrib_backwardCompat: WriteOldStyleAttribFile = 0,
KeepOldAttribFiles = 0
BackupPC_backupDelete: removing #271/Homes
__bpc_progress_state__ delete #271/Homes
BackupPC_backupDelete: No prior backup for merge
bpc_attrib_dirWrite: old attrib has same digest; no changes to ref
counts
__bpc_pidStart__ 20861
BackupPC_refCountUpdate: computing totals for host kestrel
__bpc_progress_state__ sumUpdate 0/128
bpc_attrib_backwardCompat: WriteOldStyleAttribFile = 0,
KeepOldAttribFiles = 0
__bpc_progress_state__ sumUpdate 8/128
__bpc_progress_state__ sumUpdate 16/128
__bpc_progress_state__ sumUpdate 24/128
__bpc_progress_state__ sumUpdate 32/128
__bpc_progress_state__ sumUpdate 40/128
__bpc_progress_state__ sumUpdate 48/128
__bpc_progress_state__ sumUpdate 56/128
__bpc_progress_state__ sumUpdate 64/128
__bpc_progress_state__ sumUpdate 72/128
__bpc_progress_state__ sumUpdate 80/128
__bpc_progress_state__ sumUpdate 88/128
__bpc_progress_state__ sumUpdate 96/128
__bpc_progress_state__ sumUpdate 104/128
__bpc_progress_state__ sumUpdate 112/128
__bpc_progress_state__ sumUpdate 120/128
__bpc_progress_state__ rename total
BackupPC_refCountUpdate: host kestrel got 0 errors (took 6 secs)
BackupPC_refCountUpdate total errors: 0
__bpc_pidEnd__ 20861
BackupPC_backupDelete: got 0 errors
__bpc_pidEnd__ 20851
tornado:/var/lib/backuppc/pc/kestrel# >>>
8<----------------------------------------------------------------------
After a couple of attempts to delete these directories, the next backup
of
home directories on the _local_ machine seems to have started from
scratch
five days ago and is still in progress. The actual disc usage as
reported
by the OS (df) isn't changing, but BackupPC appears to think that it
has
now used 5TB of a 3TB disc and the claimed usage is growing. If the
graph
on the status page were to be believed (and clearly is it not) the V3
pool
is growing by hundreds of gigabytes per day, and the V4 pool is
shrinking.
Output from top:
8<----------------------------------------------------------------------
tornado:~# >>> top -b -n1 -u backuppc -c
top - 12:31:18 up 266 days, 23:46, 9 users, load average: 0.60, 0.84,
0.94
Tasks: 243 total, 1 running, 241 sleeping, 0 stopped, 1 zombie
%Cpu(s): 7.2 us, 6.0 sy, 3.6 ni, 77.6 id, 4.6 wa, 0.0 hi, 1.0 si,
0.0 st
KiB Mem : 6129868 total, 329516 free, 1121492 used, 4678860
buff/cache
KiB Swap: 12582908 total, 12309444 free, 273464 used. 4750136 avail
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
18621 backuppc 39 19 85504 24340 5144 S 0.0 0.4 0:00.55
/usr/bin/perl /usr/share/backuppc/bin/BackupPC_dump localhost
18841 backuppc 39 19 0 0 0 Z 0.0 0.0 0:02.49
[rsync_bpc] <defunct>
18843 backuppc 39 19 70964 10252 224 S 0.0 0.2 0:00.63
/usr/local/bin/rsync_bpc --bpc-top-dir /var/lib/backuppc
--bpc-host-name localhost --bpc-share-name Homes --bpc-bkup-num +
19235 backuppc 39 19 57424 7568 2840 S 0.0 0.1 2:07.82
/usr/bin/perl /usr/share/backuppc/bin/BackupPC -d
8<----------------------------------------------------------------------
Cut'n'pasted from the Server Status page, note "Pool is
663.71+4946.35GiB"
8<----------------------------------------------------------------------
BackupPC Server Status
Currently Running Jobs
Host Type User Start Time Command PID Xfer PID
Status Count
localhost incr backuppc 2018-09-09 20:00 BackupPC_dump
localhost
18621 18841, 18843 backup share "Homes" 25601
General Server Information
The servers PID is 19235, on host tornado, version 4.2.1, started
at 2018-09-04 16:48.
This status was generated at 2018-09-14 11:15.
The configuration was last loaded at 2018-09-04 16:48.
PCs will be next queued at 2018-09-14 12:00.
Other info:
1 pending backup requests from last scheduled wakeup,
0 pending user backup requests,
0 pending command requests,
Pool is 663.71+4946.35GiB comprising 1675170+7424490 files and
16512+21845 directories (as of 2018-09-10 01:25),
Pool hashing gives 2220+9555 repeated files with longest chain
7+4,
Nightly cleanup removed 88290+105 files of size 580.84+1.00GiB
(around 2018-09-10 01:25),
Pool file system was recently at 80% (2018-09-14 11:15),
today's max is 80% (2018-09-14 01:00) and yesterday's max was 80%.
8<----------------------------------------------------------------------
At this point I have no confidence in BackupPC version 4.
Given what I've done, is any of this to be expected?
Yes, it is absolutely to be expected, as the tool does exactly what it
says it does. It does not do what you want it to do, which is better
accomplished in a different way.
You can, of course, delete the entire backup, then set excludes, and
back up what you want. This is probably simplest if you're going to
fool around at the storage level. I'm not entirely sure I'd recommend
it.
Is there a way to go back to V3.x?
Not directly. Of course, you can delete and reinstall. I highly
recommend you read up on how to do so, before executing a command that
does something other than what you intend, though.
I see on github that there were some changes which might affect systems
with mixed V3/V4 backups, in particular
https://u2182357.ct.sendgrid.net/wf/click?upn=rBK8reUlX8Sxr7Iz1fV-2F7dCyrjIWd3LVbeqZFg4ygUHrPSofxEWNMS5nmMh1XJxPiXGjwP3CiovdHUPS8OFRo4fwbk2qMIfu4K8yMvqARWW0NEKcRdJPR2xht4mbl8Ly_OypFYCWzG5ApGW-2FFpGTxc4RCS9eud0Dl1htN5rYoUZ8To4zeNUFBkAGI3hzer91CKYVXxkgbi31zuEYoC4RWvY-2BYjRMJVneQuyzlbo0PvdIVfsNBUnR19dvrMumhplA7GWZ18QlTnK-2FEkxl-2B8mRZBMBK1R5o0btRBLptM-2Fa5K5Wa0-2Fx0sMT-2FLcM7AAYKukkzMoqhjHeAr9e7CJX4Xqxcc7EMjViK-2BNm-2BDgU1PGjwYeo5gYHnvEZuGasNYIQMX-2Fq6
Will 4.2.2 likely improve things?
No. It's not a bad idea for its own sake, but it is unrelated to your
issue.
Any other suggestions welcome.
At a high level, don't try to screw around with the contents of
individual backups. It's possible, but you'll need to have a deep
understanding of what tools to use and the ramifications of using them.
_______________________________________________
BackupPC-users mailing list
[email protected]
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
