On Tue, 02 Jun 2015 14:02:37 -0400 Matthew Gabeler-Lee <chee...@fastcat.org>
wrote:
Package: lvm2
Version: 2.02.111-2.2
Severity: important
Last night lvm2 tried to murder my server.
OK, exaggerating a little bit, but it did something that seems not only
wrong, but quite bad.
The system was running a backup tool, using LVM snapshots to get coherent
views of the filesystem.
The snapshot for /var filled up.
LVM decided to forcibly unmount the snapshot, which I guess is OK based on
the information I've found on why this behavior was introduced, but it also
forcibly unmounted the main device, resulting in the sudden and
unceremonious loss of /var on the server, which had many unpleasant effects.
Jun 2 01:32:32 cheetah kernel: [5957103.322426] device-mapper: snapshots:
Invalidating snapshot: Unable to allocate exception.
Jun 2 01:32:33 cheetah lvm[9693]: Unmounting invalid snapshot raid5-var--snap
from /mnt/runbackup.8194.28259/var.
Jun 2 01:32:33 cheetah lvm[9693]: Unmounting invalid snapshot raid5-var--snap
from /var.
It looks like something got quite confused as to what device was backing
/var.
This doesn't look to me like lvm2 bug.
Unless raid5-var/snap was not the volume mounted on /var ??
But lvm2 does umount based on major:minor pair - so wasn't /var incorrectly
mounted from a snapshot instead of an /var origin device ?
The BZ then needs to show the table & mount state before and after.
Old snapshots are invalidated when they get full.
Such INVALID device cannot be used anymore - it's simply lost.
So whoever uses something there (.e.g. mounted fs) it is already doomed at
this moment.
If users wants to continue to use origin & snapshots - I'd probably recommend
to use thin-provisioning - which is designed for such use case much better.
If user still want to use the old snapshot for this let him configure
dmeventd monitoring of used space in snapshot - so there is some chance the
system may resize COW space in case the used space there gets above configured
threshold as the COW size is resized (when there is some free space in VG of
course).
Regards
Zdenek
