I'm using Amanda 3.2.1 on FreeBSD 8.1 and last night I did a stoopid thing with my tape changer which caused Amanda to fall into degraded mode. I found a nit and then a problem where none of my backups were done.
First the nit. From amreport: The next 2 tapes Amanda expects to use are: daily-028, daily-029. FAILURE DUMP SUMMARY: taper: FATAL Can't locate auto/Amanda/Device/Device/swig_volume.al in @INC (@INC contains: @amperldir@ /usr/local/lib/perl5/site_perl/5.10.1 /usr/local/lib/perl5/5.10.1/BSDPAN /usr/local/lib/perl5/site_perl/5.10.1/mach /usr/local/lib/perl5/5.10.1/mach /usr/local/lib/perl5/5.10.1 .) at /usr/local/lib/perl5/site_perl/5.10.1/Amanda/Device.pm line 34 Which sent me down the rabbit hole looking at Perl problems until I looked at the taper debug file: Wed Jan 19 23:00:47 2011: taper: pid 70229 ruid 501 euid 501 version 3.2.1: start at Wed Jan 19 23:00:47 2011 Wed Jan 19 23:00:47 2011: taper: pid 70229 ruid 501 euid 501 version 3.2.1: rename at Wed Jan 19 23:00:47 2011 Wed Jan 19 23:00:47 2011: taper: chg-robot: using statefile '/var/amanda/changer.daily.status' Wed Jan 19 23:00:47 2011: taper: Amanda::Taper::Scan::traditional stage 1: search for oldest reusable volume Wed Jan 19 23:00:47 2011: taper: Amanda::Taper::Scan::traditional oldest reusable volume is 'daily-028' Wed Jan 19 23:00:47 2011: taper: Amanda::Taper::Scan::traditional stage 1: searching oldest reusable volume 'daily-028' Wed Jan 19 23:00:47 2011: taper: invoking /usr/local/sbin/mtx -f /dev/daily-changer status Wed Jan 19 23:00:48 2011: taper: daily: updating state Wed Jan 19 23:00:48 2011: taper: warning: mtx indicates tape in drive 0 should go to slot 28, but that slot is not empty. Wed Jan 19 23:00:48 2011: taper: daily: loading label 'daily-028' Wed Jan 19 23:00:48 2011: taper: daily: requested volume is already in drive 0 Wed Jan 19 23:00:48 2011: taper: daily: polling 'tape:/dev/daily-tape' to see if it's ready Wed Jan 19 23:01:27 2011: taper: Device tape:/dev/daily-tape error = 'No tapestart header -- unlabeled device?' Wed Jan 19 23:01:27 2011: taper: Device tape:/dev/daily-tape setting status flag(s): DEVICE_STATUS_VOLUME_UNLABELED Wed Jan 19 23:01:27 2011: taper: daily: Expected label 'daily-028', but got an unlabeled tape Wed Jan 19 23:01:27 2011: taper: critical (fatal): Can't locate auto/Amanda/Device/Device/swig_volume.al in @INC (@INC contains: @amperldir@ /usr/local/lib/perl5/site_perl/5.10.1 /usr/local/lib/perl5/5.10.1/BSDPAN /usr/local/lib/perl5/site_perl/5.10.1/mach /usr/local/lib/perl5/5.10.1/mach /usr/local/lib/perl5/5.10.1 .) at /usr/local/lib/perl5/site_perl/5.10.1/Amanda/Device.pm line 34 It would seem that the error "Can't locate auto/Amanda/Device/Device/swig_volume.al" means that it had problems reading the tape? OK, on to the more serious problem: When it couldn't find a tape Amanda went into degraded, but then each DLE failed with: alcor / lev 0 FAILED [out of holding space in degraded mode] I've got reserve set to 80 and use to -25 GB and the holding space filesystem has 2.1 TB available, and since a usual daily backup is nearly always less than 200 GB, and never more than 400 GB, I would seem to have enough holding space. :) In the amdump log file are these two lines: driver: adding holding disk 0 dir /local-project/tmp/amanda/holding/daily size 2197978624 chunksize 9007199254740896 reserving 0 out of 2197978624 for degraded-mode dumps which clearly suggests something is wrong. I put the amdump and log files from the run, along with all the debug files at: http://www.meridian-enviro.com/rand/amanda/daily/ And yes, Jean-Louis, I even checked this time that they are readable! :) Here is the config: org "Meridian" dumpcycle 1 weeks # Do a full backup once a week runspercycle 6 days # 'daily' runs 6 times a week (Mon - Sat) runtapes 2 # Use two tape per 'daily' run tapecycle 60 tapes # 60 total tapes in the 'daily' inventory flush-threshold-dumped 700 flush-threshold-scheduled 700 taperflush 700 autoflush yes tapetype SDX-700C-AIT3-parts tapedev "daily" labelstr "^daily-[0-9][0-9]*$" infofile "/local-project/amanda/info/daily" indexdir "/local-project/amanda/index/daily" logdir "/var/log/amanda/daily" holdingdisk daily { comment "Daily holding disk" directory "/local-project/tmp/amanda/holding/daily" use -25 gbyte chunksize 0 gbyte } includefile "/usr/local/etc/amanda/amanda-changer.conf" includefile "/usr/local/etc/amanda/amanda-debug.conf" includefile "/usr/local/etc/amanda/amanda-defaults.conf" includefile "/usr/local/etc/amanda/amanda-tapetypes.conf" includefile "/usr/local/etc/amanda/amanda-dumptypes.conf" includefile "/usr/local/etc/amanda/amanda-interfaces.conf" ## ## The includefile contents are here: ## ## amanda-changer.conf define changer daily { tapedev "chg-robot:/dev/daily-changer" property "tape-device" "0=tape:/dev/daily-tape" changerfile"/var/amanda/changer.daily.status" } define changer offsite { tapedev "chg-robot:/dev/offsite-changer" property "tape-device" "0=tape:/dev/offsite-tape" changerfile"/var/amanda/changer.offsite.status" } ## amanda-defaults.conf mailto "backup" dumpuser "backup" printer "lp" usetimestamps yes netusage 1000 mbps inparallel 20 device-output-buffer-size 5 mb dumporder "BTBTBTBTBTSSSSSSSSSs" taperalgo largest bumpsize 100 mb bumpmult 2 bumpdays 3 days reserve 80 displayunit "m" dtimeout 1800 etimeout 300 columnspec "HostName=0:10,Disk=1:24,Level=1:2,OrigKB=1:8,OutKB=1:8,Compress=1:5,DumpTime=1:6,DumpRate=1:6,TapeTime=1:5,TapeRate=1:7" ## amanda-tapetypes.conf define tapetype SDX-700C-AIT3-parts { comment "Sony LIB-152E AIT-3 changer with 230m tapes" length 96000 mbytes # Supposedly a 100 GB (native) tape filemark 5637 kbytes speed 12000 kps # 12 MB/s (native) transfer rate part-size 20 gb part-cache-type disk part-cache-dir "/local-project/tmp/amanda/part-cache/daily" } define tapetype SDX-700C-AIT3 { comment "Sony LIB-152E AIT-3 changer with 230m tapes" length 96000 mbytes # Supposedly a 100 GB (native) tape filemark 5637 kbytes speed 12000 kps # 12 MB/s (native) transfer rate } define tapetype SDX-500C-230m { comment "Sony LIB-152E AIT-2 changer with 230m tapes" length 48000 mbytes # Supposedly a 50 GB (native) tape filemark 5637 kbytes speed 6144 kps # 6 MB/s (native) transfer rate } define tapetype HP-C5713A-150m { comment "Hewlett-Packard C5713a DDS-4 changer with 150 meter tapes" length 19500 mbytes # Supposedly a 20 GB (native) tape filemark 0 kbytes speed 3072 kbps # 3 MB/s (native) transfer rate } define tapetype HP-C5713A-125m { comment "Hewlett-Packard C5713A DDS-4 changer with 125 meter tapes" length 11500 mbytes # Supposedly a 12 GB (native) tape filemark 0 kbytes speed 3072 kbps # 3 MB/s (native) transfer rate } define tapetype HP-C5713A-120m { comment "Hewlett-Packard C5713A DDS-4 changer with 120 meter tapes" length 3500 mbytes # Supposedly a 4 GB (native) tape filemark 0 kbytes speed 500 kbps } define tapetype HP-C1557A-125m { comment "Hewlett-Packard C1557A DDS-3 changer with 125 meter tapes" length 11500 mbytes # Supposedly a 12 GB (native) tape filemark 0 kbytes speed 1024 kbps } define tapetype HP-C1557A-120m { comment "Hewlett-Packard C1557A pretending to be a DDS-2 with 120 meter tapes" length 3500 mbytes # Supposedly a 4 GB (native) tape filemark 100 kbytes speed 500 kbps } define tapetype HP-C1557A-90m { comment "Hewlett-Packard C1557A pretending to be a DDS-2 DAT with 90 meter tapes" length 1900 mbytes # Supposedly a 2 GB (native) tape filemark 100 kbytes speed 500 kbps } define tapetype SDT-7000-120m { comment "Sony SDT-7000 DDS-2 DAT with 120 meter tapes" length 3500 mbytes # Supposedly a 4 GB (native) tape filemark 100 kbytes speed 750 kbps } define tapetype SDT-7000-90m { comment "Sony SDT-7000 DDS-2 DAT with 90 meter tapes" length 1900 mbytes # Supposedly a 2 GB (native) tape filemark 100 kbytes speed 750 kbps } define tapetype SDT-5000-120m { comment "Sony SDT-5000 DDS-2 DAT with 120 meter tapes" length 1200 mbytes # Supposedly a 4 GB tape ?? filemark 100 kbytes speed 115 kbps } define tapetype SDT-5000-90m { comment "Sony SDT-5000 DDS-2 DAT with 90 meter tapes" length 1850 mbytes # Supposedly a 2 GB tape filemark 100 kbytes speed 350 kbps } define tapetype SDT-5000-60m { comment "Sony SDT-5000 DDS-2 DAT with 60 meter tapes" length 1200 mbytes # Supposedly a 1.3 GB tape filemark 100 kbytes speed 350 kbps } define tapetype DISK { comment "Dump to local disk for testing." length 10240 mbytes } ## amanda-dumptypes.conf define dumptype defaults { comment "Global defaults for dump types" auth "BSD" compress client fast comprate 0.60, 0.60 # Be somewhat pessemestic about guesses estimate SERVER index yes holdingdisk yes } define dumptype dump { defaults auth "bsdtcp" program "DUMP" } define dumptype offdump { defaults auth "bsdtcp" dumpcycle 0 program "DUMP" } define dumptype tcp { defaults auth "bsdtcp" program "GNUTAR" exclude list ".amanda.excludes" } define dumptype tar { defaults program "GNUTAR" exclude list ".amanda.excludes" } define dumptype offsite { defaults program "GNUTAR" record no dumpcycle 0 exclude list ".amanda.excludes" } define dumptype offtcp { defaults auth "bsdtcp" program "GNUTAR" record no dumpcycle 0 exclude list ".amanda.excludes" } define dumptype ssh { defaults auth "ssh" ssh-keys "/var/amanda/.ssh/id_rsa" client-username "backup" } define dumptype daily-tar-ssh { ssh program "GNUTAR" exclude list optional ".amanda.excludes" } define dumptype offsite-tar-ssh { ssh program "GNUTAR" record no dumpcycle 0 exclude list optional ".amanda.excludes" } ## amanda-interfaces define interface local { comment "A local disk" use 10000 mbps } define interface dc0 { comment "100 Mbps Ethernet" use 10000 mbps }
