** Description changed: - At least four users, including myself, are having an issue with update- - initramfs hanging while updating ubuntu 16.04. The bug has been - documented while attempting an update to multiple kernel versions ( - 4.4.0-24, 4.4.0-62, 4.4.0-63). The bug causes any apt-get update or - install to fail, and may also lead to an unbootable system. + [Impact] + If there is lots of activity on a different filesystem, high io/cache pressure, or a stale mount updating initramfs might hang because the update-initramfs script calls sync. This leaves the system unresponsive and will require the user to forcefully kill dpkg. + + [Test Case] + One easy way to simulate this is by copying a huge file over nfs. Note: do not use localhost, it is too fast for this purpose, a network bridge over qemu is good enough. + + 1. Setup a nfs server and export a directory with write permissions for + the client + + 2. Copy a big file over NFS (assumes it is mounted at /mnt in the client) + $ dd if=/dev/zero of=/mnt/big_zeros bs=512 count=1M status=progress + Note: adjust the "count" accordingly to how fast/slow your system is. + + 3. Call sync and verify that it only finished after "dd" is done. + + 4. Repeat step #2 and now call "sync -f /boot/<some-initrd-image>". + Verify that it finished immediatly. + + [Regression Potential] + Although syncfs(2) man page [2] states that it works just the same as sync - except being limited to a filesystem -, any actual differences in the implementation can affect how the initrd image file is synced to the disk, leading to unexpected behavior. + + [Other] + If syncfs is unsupported then the 'sync' binary will ensure that a normal system-wide sync is called [1]. The proposed patch won't help in scenarios where the initrd file is located in a busy filesystem (eg. no separated /boot partition). + + The 'sync' and 'syncfs' specifications are pretty much similar and + syncfs is expected to work the same as a normal sync regarding data + integrity. Also altought the patch introduces the initrd file as an + extra argument which would cause 'sync -f' to fail if the file does not + exist, the update-initramfs script would fail earlier than that because + that file is created by the 'mv' call a few lines above. Thus no + regressions are expected. + + [References] + [1] http://git.savannah.gnu.org/cgit/coreutils.git/commit/src/sync.c?id=8b2bf5295f353016d4f5e6a2317d55b6a8e7fd00 + [2] http://manpages.ubuntu.com/manpages/xenial/man2/syncfs.2.html + + [Original bug report] + At least four users, including myself, are having an issue with update-initramfs hanging while updating ubuntu 16.04. The bug has been documented while attempting an update to multiple kernel versions ( 4.4.0-24, 4.4.0-62, 4.4.0-63). The bug causes any apt-get update or install to fail, and may also lead to an unbootable system. User #1 (me): $ uname -a Linux <computername> 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux $ sudo apt-get upgrade - Fetched 1,571 MB in 2min 9s (12.2 MB/s) + Fetched 1,571 MB in 2min 9s (12.2 MB/s) Extracting templates from packages: 100% Preconfiguring packages ... (Reading database ... 344634 files and directories currently installed.) Preparing to unpack .../base-files_9.4ubuntu4.4_amd64.deb ... Unpacking base-files (9.4ubuntu4.4) over (9.4ubuntu4.3) ... Processing triggers for plymouth-theme-ubuntu-text (0.9.2-3ubuntu13.1) ... update-initramfs: deferring update (trigger activated) Processing triggers for man-db (2.7.5-1) ... Processing triggers for cracklib-runtime (2.9.2-1build2) ... Processing triggers for install-info (6.1.0.dfsg.1-5) ... Processing triggers for initramfs-tools (0.122ubuntu8.8) ... update-initramfs: Generating /boot/initrd.img-4.4.0-63-generic <hangs overnight> Uh oh! Let's try to troubleshoot. $ sudo killall dpkg $ sudo dpkg --configure -a Setting up initramfs-tools (0.122ubuntu8.8) ... update-initramfs: deferring update (trigger activated) Setting up base-files (9.4ubuntu4.4) ... Installing new version of config file /etc/issue ... Installing new version of config file /etc/issue.net ... Installing new version of config file /etc/lsb-release ... Setting up linux-image-4.4.0-63-generic (4.4.0-63.84) ... Running depmod. update-initramfs: deferring update (hook will be called later) initrd.img(/boot/initrd.img-4.4.0-63-generic ) points to /boot/initrd.img-4.4.0-63-generic - (/boot/initrd.img-4.4.0-63-generic) -- doing nothing at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 491. + (/boot/initrd.img-4.4.0-63-generic) -- doing nothing at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 491. vmlinuz(/boot/vmlinuz-4.4.0-63-generic ) points to /boot/vmlinuz-4.4.0-63-generic - (/boot/vmlinuz-4.4.0-63-generic) -- doing nothing at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 491. + (/boot/vmlinuz-4.4.0-63-generic) -- doing nothing at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 491. Examining /etc/kernel/postinst.d. run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-63-generic /boot/vmlinuz-4.4.0-63-generic run-parts: executing /etc/kernel/postinst.d/dkms 4.4.0-63-generic /boot/vmlinuz-4.4.0-63-generic run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-63-generic /boot/vmlinuz-4.4.0-63-generic update-initramfs: Generating /boot/initrd.img-4.4.0-63-generic ^CFailed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 1052. dpkg: error processing package linux-image-4.4.0-63-generic (--configure): - subprocess installed post-installation script returned error exit status 2 + subprocess installed post-installation script returned error exit status 2 dpkg: dependency problems prevent configuration of linux-image-generic: - linux-image-generic depends on linux-image-4.4.0-63-generic; however: - Package linux-image-4.4.0-63-generic is not configured yet. + linux-image-generic depends on linux-image-4.4.0-63-generic; however: + Package linux-image-4.4.0-63-generic is not configured yet. dpkg: error processing package linux-image-generic (--configure): - dependency problems - leaving unconfigured + dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of linux-signed-image-4.4.0-63-generic: - linux-signed-image-4.4.0-63-generic depends on linux-image-4.4.0-63-generic (= 4.4.0-63.84); however: - Package linux-image-4.4.0-63-generic is not configured yet. + linux-signed-image-4.4.0-63-generic depends on linux-image-4.4.0-63-generic (= 4.4.0-63.84); however: + Package linux-image-4.4.0-63-generic is not configured yet. dpkg: error processing package linux-signed-image-4.4.0-63-generic (--configure): - dependency problems - leaving unconfigured + dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of linux-signed-image-generic: - linux-signed-image-generic depends on linux-signed-image-4.4.0-63-generic; however: - Package linux-signed-image-4.4.0-63-generic is not configured yet. + linux-signed-image-generic depends on linux-signed-image-4.4.0-63-generic; however: + Package linux-signed-image-4.4.0-63-generic is not configured yet. dpkg: error processing package linux-signed-image-generic (--configure): - dependency problems - leaving unconfigured + dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of linux-image-extra-4.4.0-63-generic: - linux-image-extra-4.4.0-63-generic depends on linux-image-4.4.0-63-generic; however: - Package linux-image-4.4.0-63-generic is not configured yet. + linux-image-extra-4.4.0-63-generic depends on linux-image-4.4.0-63-generic; however: + Package linux-image-4.4.0-63-generic is not configured yet. dpkg: error processing package linux-image-extra-4.4.0-63-generic (--configure): - dependency problems - leaving unconfigured + dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of linux-generic: - linux-generic depends on linux-image-generic (= 4.4.0.63.67); however: - Package linux-image-generic is not configured yet. + linux-generic depends on linux-image-generic (= 4.4.0.63.67); however: + Package linux-image-generic is not configured yet. dpkg: error processing package linux-generic (--configure): - dependency problems - leaving unconfigured + dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of linux-signed-generic: - linux-signed-generic depends on linux-signed-image-generic (= 4.4.0.63.67); however: - Package linux-signed-image-generic is not configured yet. + linux-signed-generic depends on linux-signed-image-generic (= 4.4.0.63.67); however: + Package linux-signed-image-generic is not configured yet. dpkg: error processing package linux-signed-generic (--configure): - dependency problems - leaving unconfigured + dependency problems - leaving unconfigured Processing triggers for initramfs-tools (0.122ubuntu8.8) ... update-initramfs: Generating /boot/initrd.img-4.4.0-63-generic ^Cdpkg: error processing package initramfs-tools (--configure): - subprocess installed post-installation script was interrupted + subprocess installed post-installation script was interrupted Processing triggers for plymouth-theme-ubuntu-text (0.9.2-3ubuntu13.1) ... update-initramfs: deferring update (trigger activated) Errors were encountered while processing: - linux-image-4.4.0-63-generic - linux-image-generic - linux-signed-image-4.4.0-63-generic - linux-signed-image-generic - linux-image-extra-4.4.0-63-generic - linux-generic - linux-signed-generic - initramfs-tools - + linux-image-4.4.0-63-generic + linux-image-generic + linux-signed-image-4.4.0-63-generic + linux-signed-image-generic + linux-image-extra-4.4.0-63-generic + linux-generic + linux-signed-generic + initramfs-tools I verified that /boot has enough space: /dev/nvme0n1p2 496M 314M 157M 67% /boot Here's my /boot: $ ls -l /boot total 325378 -rw-r--r-- 1 root root 1244118 Jan 6 16:44 abi-4.4.0-59-generic -rw-r--r-- 1 root root 1244118 Jan 18 07:59 abi-4.4.0-62-generic -rw-r--r-- 1 root root 1245512 Feb 1 11:39 abi-4.4.0-63-generic -rw-r--r-- 1 root root 1245512 Feb 20 05:40 abi-4.4.0-64-generic -rw-r--r-- 1 root root 190047 Jan 6 16:44 config-4.4.0-59-generic -rw-r--r-- 1 root root 190047 Jan 18 07:59 config-4.4.0-62-generic -rw-r--r-- 1 root root 190247 Feb 1 11:39 config-4.4.0-63-generic -rw-r--r-- 1 root root 190247 Feb 20 05:40 config-4.4.0-64-generic drwx------ 3 root root 4096 Dec 31 1969 efi drwxr-xr-x 5 root root 1024 Feb 23 13:23 grub -rw-r--r-- 1 root root 10931153 Feb 23 13:56 initrd.img-4.4.0-57-generic -rw-r--r-- 1 root root 46506399 Feb 9 05:37 initrd.img-4.4.0-59-generic -rw-r--r-- 2 root root 39526162 Feb 23 13:27 initrd.img-4.4.0-62-generic -rw-r--r-- 2 root root 39526162 Feb 23 13:27 initrd.img-4.4.0-62-generic.dpkg-bak -rw-r--r-- 1 root root 0 Feb 23 13:56 initrd.img-4.4.0-62-generic.new -rw-r--r-- 2 root root 26019417 Feb 23 13:11 initrd.img-4.4.0-63-generic -rw-r--r-- 2 root root 26019417 Feb 23 13:11 initrd.img-4.4.0-63-generic.dpkg-bak -rw-r--r-- 1 root root 0 Feb 23 13:56 initrd.img-4.4.0-63-generic.new -rw-r--r-- 1 root root 39516686 Feb 23 14:50 initrd.img-4.4.0-64-generic -rw-r--r-- 1 root root 39516505 Feb 23 13:33 initrd.img-4.4.0-64-generic.dpkg-bak drwx------ 2 root root 12288 Jul 31 2016 lost+found -rw-r--r-- 1 root root 182704 Jan 28 2016 memtest86+.bin -rw-r--r-- 1 root root 184380 Jan 28 2016 memtest86+.elf -rw-r--r-- 1 root root 184840 Jan 28 2016 memtest86+_multiboot.bin -rw------- 1 root root 3875594 Jan 6 16:44 System.map-4.4.0-59-generic -rw------- 1 root root 3875553 Jan 18 07:59 System.map-4.4.0-62-generic -rw------- 1 root root 3883990 Feb 1 11:39 System.map-4.4.0-63-generic -rw------- 1 root root 3883990 Feb 20 05:40 System.map-4.4.0-64-generic -rw------- 1 root root 7069136 Jan 6 16:44 vmlinuz-4.4.0-59-generic -rw------- 1 root root 7071064 Jan 11 07:08 vmlinuz-4.4.0-59-generic.efi.signed -rw------- 1 root root 7070992 Jan 18 07:59 vmlinuz-4.4.0-62-generic -rw------- 1 root root 7072920 Feb 3 08:04 vmlinuz-4.4.0-62-generic.efi.signed -rw------- 1 root root 7087088 Feb 1 11:39 vmlinuz-4.4.0-63-generic -rw------- 1 root root 7087152 Feb 20 05:40 vmlinuz-4.4.0-64-generic After uninstall of linux-image-4.4.0-63-generic: $ sudo dpkg --status linux-image-4.4.0-63-genericPackage: linux-image-4.4.0-63-generic Status: deinstall ok config-files Priority: optional Section: kernel Installed-Size: 64800 Maintainer: Ubuntu Kernel Team <kernel-t...@lists.ubuntu.com> Architecture: amd64 Source: linux Version: 4.4.0-63.84 Provides: fuse-module, ivtv-modules, kvm-api-4, linux-image, redhat-cluster-modules, spl-dkms, virtualbox-guest-modules, zfs-dkms Depends: initramfs-tools | linux-initramfs-tool, kmod Recommends: grub-pc | grub-efi-amd64 | grub-efi-ia32 | grub | lilo Suggests: fdutils, linux-doc-4.4.0 | linux-source-4.4.0, linux-tools, linux-headers-4.4.0-63-generic There does seem to be a problem with linux-image-extra-4.4.0-63-generic. Trying to uninstall this triggers update-initramfs. $ sudo dpkg --status linux-image-extra-4.4.0-63-generic Package: linux-image-extra-4.4.0-63-generic Status: purge ok half-installed Priority: optional Section: kernel Installed-Size: 148939 Maintainer: Ubuntu Kernel Team <kernel-t...@lists.ubuntu.com> Architecture: amd64 Source: linux Version: 4.4.0-63.84 Depends: linux-image-4.4.0-63-generic, crda | wireless-crda I've tried http://askubuntu.com/questions/650732/apt-fails-to-remove- partially-installed-kernel-and-cant-install-any-other-packa/651596 with no luck. At this point I'm not sure if a partially installed kernel is the problem, and I've run out of troubleshooting insight. - User #2: https://ubuntuforums.org/showthread.php?t=2351895 User #3: https://ubuntuforums.org/showthread.php?t=2353606 User #4 (original question): When upgrading (apt-get upgrade) on 16.04 (ubuntu-mate) I get a hang (left from 9:50am til 16:00 or thereabouts) I tracked down as far as the call to /etc/kernel/postinst.d/initramfs- tools and added a -v option, the last relevant lines on the terminal when re-running `apt-get upgrade` are as follows: ... Adding binary /bin/date Adding binary /sbin/hwclock Adding binary /sbin/dumpe2fs Calling hook dmsetup Calling hook reiserfsprogs Building cpio /boot/initrd.img-4.4.0-24-generic.new initramfs ^CFailed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-24-generic.postinst line 1052. dpkg: error processing package linux-image-4.4.0-24-generic (--configure): - subprocess installed post-installation script returned error exit status 2 + subprocess installed post-installation script returned error exit status 2 dpkg: dependency problems prevent configuration of linux-image-extra-4.4.0-24-generic: - linux-image-extra-4.4.0-24-generic depends on linux-image-4.4.0-24-generic; however: - Package linux-image-4.4.0-24-generic is not configured yet. - - - Checking with `ps fax` before the interrupt, I get this tree - indicating a hang on `sync` task + linux-image-extra-4.4.0-24-generic depends on linux-image-4.4.0-24-generic; however: + Package linux-image-4.4.0-24-generic is not configured yet. + + Checking with `ps fax` before the interrupt, I get this tree - + indicating a hang on `sync` task sudo dpkg --configure -a - \_ dpkg --configure -a - \_ /usr/bin/perl /var/lib/dpkg/info/linux-image-4.4.0-24-generic.postinst configure - \_ run-parts --verbose --exit-on-error --arg=4.4.0-24-generic --arg=/boot/vmlinuz-4.4.0-24-generic /etc/kernel/postinst.d - \_ /bin/sh -e /etc/kernel/postinst.d/initramfs-tools 4.4.0-24-generic /boot/vmlinuz-4.4.0-24-generic - \_ /bin/sh /usr/sbin/update-initramfs -v -c -t -k 4.4.0-24-generic -b /boot - \_ sync + \_ dpkg --configure -a + \_ /usr/bin/perl /var/lib/dpkg/info/linux-image-4.4.0-24-generic.postinst configure + \_ run-parts --verbose --exit-on-error --arg=4.4.0-24-generic --arg=/boot/vmlinuz-4.4.0-24-generic /etc/kernel/postinst.d + \_ /bin/sh -e /etc/kernel/postinst.d/initramfs-tools 4.4.0-24-generic /boot/vmlinuz-4.4.0-24-generic + \_ /bin/sh /usr/sbin/update-initramfs -v -c -t -k 4.4.0-24-generic -b /boot + \_ sync I tried moving the resulting initramfs img file in /boot so a new one would be created - a new one is indeed created, hang still occurs on `sync`
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1667512 Title: update-initramfs hangs on upgrade, dpkg unusable, unbootable system To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1667512/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
