Public bug reported: Starting on Noble, we see /boot fail to mount in approximately one out of every two thousand boots. The error looks like this:
Found device dev-disk-by\x2dlabel-BOOT.device - QEMU NVMe Ctrl BOOT. Starting systemd-fsck@dev-disk-by… Check on /dev/disk/by-label/BOOT... Checking in progress on 1 disk (0.0% complete) Checking in progress on 0 disks (100.0% complete) Finished msystemd-fsck@dev-disk-by… Check on /dev/disk/by-label/BOOT. Mounting boot.mount - /boot... [ 3.051612] /dev/disk/by-label/BOOT: Can't lookup blockdev FAILED Failed to mount boot.mount - /boot. See 'systemctl status boot.mount' for details. This has resulted in a number of different failure modes for our users. Anything that needs to interact with /boot during provisioning will fail. This is usually something running update-grub or similar. If we manage to succeed in booting, this can cause subsequent kernel updates or tools that install kernel modules to fail, because update-grub fails. We've _also_ seen this manifest on the root filesystem. In that case, the boot succeeded, but the by-label links remain absent. When this occurs, we find installing kernel packages fails because mkinitramfs can't locate the root disk by label. It's a vexing problem, and so to reproduce we ran cloud images in a boot loop until we could reliably reproduce the problem. Unfortunately, we weren't able to work out anything that made this happen faster, so it's been a bit slow coming. It turns out the problem here is that libblkid recently added support to compute the checksum of the superblocks on ext4 filesystems, and Noble is the first release to include a version of util-linux new enough to have this feature. When libblkid determines an ext4 superblock's checksum is corrupt, it refuses to identify the device as having a filesystem, which leads to the removeal of the uuid and by-label fields. systemd-udevd then removes these symlinks. This is where it all goes wrong. From our debug traces, it's possible to see this clearly: (udev-worker)[208]: nvme0n1p16: Probe /dev/nvme0n1p16 with raid and offset=0 systemd-udevd[208]: 208: libblkid: LOWPROBE: [36] ext4dev: systemd-udevd[208]: 208: libblkid: BUFFER: reuse: off=1024 len=1024 (for off=1024 len=1024) systemd-udevd[208]: 208: libblkid: LOWPROBE: magic sboff=56, kboff=1 systemd-udevd[208]: 208: libblkid: LOWPROBE: call probefunc() systemd-udevd[208]: 208: libblkid: BUFFER: reuse: off=1024 len=1024 (for off=1024 len=1024) systemd-udevd[208]: incorrect checksum for type ext4dev, got D919EB5600000000, expected A47F6CF000000000 systemd-udevd[208]: 208: libblkid: LOWPROBE: [37] ext4: systemd-udevd[208]: 208: libblkid: BUFFER: reuse: off=1024 len=1024 (for off=1024 len=1024) systemd-udevd[208]: 208: libblkid: LOWPROBE: magic sboff=56, kboff=1 systemd-udevd[208]: 208: libblkid: LOWPROBE: call probefunc() systemd-udevd[208]: 208: libblkid: BUFFER: reuse: off=1024 len=1024 (for off=1024 len=1024) systemd-udevd[208]: incorrect checksum for type ext4, got D919EB5600000000, expected A47F6CF000000000 (udev-worker)[208]: nvme0n1p16: Removing/updating old device symlink '/dev/disk/by-uuid/ce209fd3-a875-4607-9089-35b9de605bd0', which is no longer belonging to this device. (udev-worker)[208]: nvme0n1p16: No reference left for '/dev/disk/by-uuid/ce209fd3-a875-4607-9089-35b9de605bd0', removing (udev-worker)[208]: nvme0n1p16: Removing/updating old device symlink '/dev/disk/by-label/BOOT', which is no longer belonging to this device. (udev-worker)[208]: nvme0n1p16: No reference left for '/dev/disk/by-label/BOOT', removing (udev-worker)[208]: nvme0n1p16: Successfully created symlink '/dev/block/259:4' to '/dev/nvme0n1p16' (udev-worker)[208]: nvme0n1p16: sd-device: Created db file '/run/udev/data/b259:4' for '/devices/pci0000:00/0000:00:01.0/nvme/nvme0/nvme0n1/nvme0n1p16' (udev-worker)[208]: nvme0n1p16: Adding watch on '/dev/nvme0n1p16' (udev-worker)[208]: nvme0n1p16: Device processed (SEQNUM=1630, ACTION=change) (udev-worker)[208]: nvme0n1p16: sd-device-monitor(worker): Passed 1315 byte to netlink monitor. Fortunately, the fix here is straight-forward and is similar to what we did for resize2fs: use O_DIRECT when reading the superblock. We've already sent a patch upstream and gotten it accepted there: https://github.com/util-linux/util- linux/commit/483c9f38e377ff0b009f546a2c4ee91a1d61588c We've also been running with a version of this patch backported to 2.39.3-9ubuntu6.1 in our production environment for a couple of months. It's completely eliminated this problem for us. We were observing this about once an hour and have had no recurrence since rolling out the fix. We had a discussion with Ted Ts'o about this as well, and he had some ideas for future improvements, but nothing that we're implementing in this fix: https://lore.kernel.org/util- linux/6d16e6d83ab48d2ea4402db17c9c0ed5514933a7.1731961869.git.k...@templeofstupid.com/T/#m55eb5087639dcfcfd5708144b1b48caf0cf762b8 I'm attaching the patch that we've applied to our tree and would be grateful if you'd pull this into util-linux >= 2.39. ** Affects: util-linux (Ubuntu) Importance: Undecided Status: New ** Patch added: "libblkid noble patch" https://bugs.launchpad.net/bugs/2090972/+attachment/5841929/+files/0001-libblkid-fix-spurious-ext-superblock-checksum-mismat.patch ** Summary changed: - libblkid: fix spurious ext superblock checksum mismatches + /boot intermittently fails to mount ** Summary changed: - /boot intermittently fails to mount + /boot intermittently fails to mount on boot -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2090972 Title: /boot intermittently fails to mount on boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/2090972/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
