I'd like to follow up because the issue seems to have cleared up for us after installing linux 5.0.1 about 40 days ago. It's hard to say whether everyone is experiencing the same bugs, but give 5.x a shot and let us know how it goes!
Just to recap. Every week or so we were seeing R/O file systems with the follow errors, which required reboot & fsck. EXT4-fs error (device vda2): ext4_mb_generate_buddy:757: group 144, block bitmap and bg descriptor inconsistent: 23914 vs 23913 free clusters Aborting journal on device vda2-8. EXT4-fs (vda2): Remounting filesystem read-only We never experienced any corruption on the host itself, only under KVM guests. Host DELL Poweredge 2950III Several KVM Guests: linux OS, distro&kernel doesn't make any difference, all randomly vulnerable during periods of high disk activity. Not sure it matters, but in our case we were using LVM2 volumes on the host and kvm media was configured as follows "media=disk,if=virtio,cache=none,aio=native,format=raw". We initially thought just one guest was affected, but over time we saw it happen with many distros and kernels. It wasn't until we had an extended period of downtime that we decided to reinstall the host with a 5.x kernel. None of the guests experienced any issues since, fingers crossed. At this point, it's hard to recommend Ubuntu 19.04 given that it's only a few months away from EOL, however the 5.x kernel seems promising whereas the Ubuntu 18.04LTS runs an older kernel that is still known to exhibit the corruption. For LTS I'd look into running it under a custom setup with a newer kernel. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-lts-utopic in Ubuntu. https://bugs.launchpad.net/bugs/1423672 Title: ext4_mb_generate_buddy:756: group N, block bitmap and bg descriptor inconsistent: X vs Y Status in Linux: Confirmed Status in linux package in Ubuntu: Confirmed Status in linux-lts-utopic package in Ubuntu: Invalid Status in linux source package in Trusty: Fix Released Status in linux-lts-utopic source package in Trusty: Fix Released Status in linux package in Debian: Fix Released Bug description: SRU Justification: Impact: Users of VMs running 3.13/3.16 and ext4 can experience data corruption in the guest. Fix: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?h=7dec5603b6b8dc4c3e1c65d318bd2a5a8c62a424 Testcase: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=818502#22 -- I noticed that one of my VM had this "dmesg -T" output: [Tue Feb 17 09:53:27 2015] systemd-udevd[5433]: starting version 204 [Thu Feb 19 06:25:08 2015] EXT4-fs error (device vda1): ext4_mb_generate_buddy:756: group 5, block bitmap and bg descriptor inconsistent: 16446 vs 16445 free clusters [Thu Feb 19 06:25:09 2015] Aborting journal on device vda1-8. [Thu Feb 19 06:25:09 2015] EXT4-fs (vda1): Remounting filesystem read-only [Thu Feb 19 06:25:09 2015] ------------[ cut here ]------------ [Thu Feb 19 06:25:09 2015] WARNING: CPU: 0 PID: 9946 at /build/buildd/linux-3.13.0/fs/ext4/ext4_jbd2.c:259 __ext4_handle_dirty_metadata+0x1a2/0x1c0() [Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy [Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Not tainted 3.13.0-45-generic #74-Ubuntu [Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 [Thu Feb 19 06:25:09 2015] 0000000000000009 ffff880003a11aa0 ffffffff81720eb6 0000000000000000 [Thu Feb 19 06:25:09 2015] ffff880003a11ad8 ffffffff810677cd ffff880000c41340 0000000000000000 [Thu Feb 19 06:25:09 2015] ffff88000a58e000 ffffffff81835280 0000000000001302 ffff880003a11ae8 [Thu Feb 19 06:25:09 2015] Call Trace: [Thu Feb 19 06:25:09 2015] [<ffffffff81720eb6>] dump_stack+0x45/0x56 [Thu Feb 19 06:25:09 2015] [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0 [Thu Feb 19 06:25:09 2015] [<ffffffff810678aa>] warn_slowpath_null+0x1a/0x20 [Thu Feb 19 06:25:09 2015] [<ffffffff8126e862>] __ext4_handle_dirty_metadata+0x1a2/0x1c0 [Thu Feb 19 06:25:09 2015] [<ffffffff81246a5a>] ? ext4_dirty_inode+0x2a/0x60 [Thu Feb 19 06:25:09 2015] [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0 [Thu Feb 19 06:25:09 2015] [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30 [Thu Feb 19 06:25:09 2015] [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0 [Thu Feb 19 06:25:09 2015] [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370 [Thu Feb 19 06:25:09 2015] [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0 [Thu Feb 19 06:25:09 2015] [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0 [Thu Feb 19 06:25:09 2015] [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0 [Thu Feb 19 06:25:09 2015] [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0 [Thu Feb 19 06:25:09 2015] [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0 [Thu Feb 19 06:25:09 2015] [<ffffffff811d8f60>] evict+0xb0/0x1b0 [Thu Feb 19 06:25:09 2015] [<ffffffff811d9775>] iput+0xf5/0x180 [Thu Feb 19 06:25:09 2015] [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200 [Thu Feb 19 06:25:09 2015] [<ffffffff811d4795>] dput+0xa5/0x180 [Thu Feb 19 06:25:09 2015] [<ffffffff811bf7e6>] __fput+0x176/0x260 [Thu Feb 19 06:25:09 2015] [<ffffffff811bf91e>] ____fput+0xe/0x10 [Thu Feb 19 06:25:09 2015] [<ffffffff810882f7>] task_work_run+0xa7/0xe0 [Thu Feb 19 06:25:09 2015] [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0 [Thu Feb 19 06:25:09 2015] [<ffffffff81731c2a>] int_signal+0x12/0x17 [Thu Feb 19 06:25:09 2015] ---[ end trace ebff9843d81b5c41 ]--- [Thu Feb 19 06:25:09 2015] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028 [Thu Feb 19 06:25:09 2015] IP: [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160 [Thu Feb 19 06:25:09 2015] PGD 167067 PUD 161067 PMD 0 [Thu Feb 19 06:25:09 2015] Oops: 0000 [#1] SMP [Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy [Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Tainted: G W 3.13.0-45-generic #74-Ubuntu [Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 [Thu Feb 19 06:25:09 2015] task: ffff880009ac4800 ti: ffff880003a10000 task.ti: ffff880003a10000 [Thu Feb 19 06:25:09 2015] RIP: 0010:[<ffffffff8125d4c1>] [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160 [Thu Feb 19 06:25:09 2015] RSP: 0000:ffff880003a11a58 EFLAGS: 00010292 [Thu Feb 19 06:25:09 2015] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000086 [Thu Feb 19 06:25:09 2015] RDX: 0000000000001302 RSI: ffffffff81a6e81f RDI: 0000000000000000 [Thu Feb 19 06:25:09 2015] RBP: ffff880003a11ae8 R08: ffffffff81a78568 R09: 0000000000000005 [Thu Feb 19 06:25:09 2015] R10: 00000000ffffffe2 R11: ffff880003a117ce R12: 0000000000000086 [Thu Feb 19 06:25:09 2015] R13: ffffffff81835280 R14: 0000000000001302 R15: ffffffff81a78568 [Thu Feb 19 06:25:09 2015] FS: 00007f74eaca4840(0000) GS:ffff88000b800000(0000) knlGS:0000000000000000 [Thu Feb 19 06:25:09 2015] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [Thu Feb 19 06:25:09 2015] CR2: 0000000001de4000 CR3: 0000000009b45000 CR4: 00000000000006f0 [Thu Feb 19 06:25:09 2015] Stack: [Thu Feb 19 06:25:09 2015] ffff880003a11a60 0000000000000103 ebff9843d81b5c41 000000000000321e [Thu Feb 19 06:25:09 2015] 00000000000014d8 0000000000000092 000000000000020e ffff88000a58e000 [Thu Feb 19 06:25:09 2015] ffff880003a11ae8 ffffffff8126e372 ffffffff810677df ffff880000c41340 [Thu Feb 19 06:25:09 2015] Call Trace: [Thu Feb 19 06:25:09 2015] [<ffffffff8126e372>] ? ext4_journal_abort_handle+0x42/0xc0 [Thu Feb 19 06:25:09 2015] [<ffffffff810677df>] ? warn_slowpath_common+0x8f/0xa0 [Thu Feb 19 06:25:09 2015] [<ffffffff8126e7cf>] __ext4_handle_dirty_metadata+0x10f/0x1c0 [Thu Feb 19 06:25:09 2015] [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0 [Thu Feb 19 06:25:09 2015] [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30 [Thu Feb 19 06:25:09 2015] [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0 [Thu Feb 19 06:25:09 2015] [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370 [Thu Feb 19 06:25:09 2015] [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0 [Thu Feb 19 06:25:09 2015] [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0 [Thu Feb 19 06:25:09 2015] [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0 [Thu Feb 19 06:25:09 2015] [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0 [Thu Feb 19 06:25:09 2015] [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0 [Thu Feb 19 06:25:09 2015] [<ffffffff811d8f60>] evict+0xb0/0x1b0 [Thu Feb 19 06:25:09 2015] [<ffffffff811d9775>] iput+0xf5/0x180 [Thu Feb 19 06:25:09 2015] [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200 [Thu Feb 19 06:25:09 2015] [<ffffffff811d4795>] dput+0xa5/0x180 [Thu Feb 19 06:25:09 2015] [<ffffffff811bf7e6>] __fput+0x176/0x260 [Thu Feb 19 06:25:09 2015] [<ffffffff811bf91e>] ____fput+0xe/0x10 [Thu Feb 19 06:25:09 2015] [<ffffffff810882f7>] task_work_run+0xa7/0xe0 [Thu Feb 19 06:25:09 2015] [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0 [Thu Feb 19 06:25:09 2015] [<ffffffff81731c2a>] int_signal+0x12/0x17 [Thu Feb 19 06:25:09 2015] Code: 48 89 e5 41 57 4d 89 c7 41 56 41 89 d6 41 55 49 89 f5 48 c7 c6 1f e8 a6 81 41 54 49 89 cc 53 48 89 fb 48 83 ec 68 4c 89 4c 24 60 <48> 8b 47 28 48 8b 57 40 48 8b 80 f8 02 00 00 48 8b 40 68 89 90 [Thu Feb 19 06:25:09 2015] RIP [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160 [Thu Feb 19 06:25:09 2015] RSP <ffff880003a11a58> [Thu Feb 19 06:25:09 2015] CR2: 0000000000000028 [Thu Feb 19 06:25:10 2015] ---[ end trace ebff9843d81b5c42 ]--- cron.daily jobs fired at 6:25:01 apparently: # tail -n2 /var/log/syslog Feb 19 06:17:01 git CRON[9848]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 19 06:25:01 git CRON[9853]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )) # run-parts --test /etc/cron.daily /etc/cron.daily/apt /etc/cron.daily/autoremove /etc/cron.daily/dpkg /etc/cron.daily/hdd-backup /etc/cron.daily/logrotate /etc/cron.daily/passwd /etc/cron.daily/upstart It seems like all the jobs ran and the upstart one somehow triggered the crash: # ls -alt /var/log/upstart/ | head total 272 drwxrwxr-x 5 root syslog 4096 Feb 19 06:25 .. drwxr-xr-x 2 root root 4096 Feb 14 06:25 . -rw-r----- 1 root root 591 Feb 12 12:26 ureadahead.log.1.gz -rw-r----- 1 root root 178 Feb 12 12:25 mountall.log.1.gz Now that I have collected some information (sorry, I don't have ubuntu-bug installed on the VM) I'll reboot it and see how it goes. More information on the VM: # lsb_release -rd Description: Ubuntu 14.04.2 LTS Release: 14.04 # apt-cache policy linux-image-3.13.0-45-generic linux-image-3.13.0-45-generic: Installed: 3.13.0-45.74 Candidate: 3.13.0-45.74 Version table: *** 3.13.0-45.74 0 500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages 100 /var/lib/dpkg/status --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Feb 19 14:34 seq crw-rw---- 1 root audio 116, 33 Feb 19 14:34 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.7 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: [Errno 2] No such file or directory CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 13.891047] init: console-font main process (855) terminated with status 71 [ 13.952825] init: plymouth-splash main process (870) terminated with status 1 [ 217.853139] random: nonblocking pool is initialized DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory Lspci: Error: [Errno 2] No such file or directory Lsusb: Error: [Errno 2] No such file or directory MachineType: QEMU Standard PC (i440FX + PIIX, 1996) Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: root=UUID=cb9cbdad-c668-4503-85db-fcf9b02f3495 ro console=tty0 console=ttyS0,38400 ProcVersionSignature: Ubuntu 3.13.0-45.74-generic 3.13.11-ckt13 RelatedPackageVersions: linux-restricted-modules-3.13.0-45-generic N/A linux-backports-modules-3.13.0-45-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty Uname: Linux 3.13.0-45-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 01/01/2011 dmi.bios.vendor: Bochs dmi.bios.version: Bochs dmi.chassis.type: 1 dmi.chassis.vendor: Bochs dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-2.0:cvnBochs:ct1:cvr: dmi.product.name: Standard PC (i440FX + PIIX, 1996) dmi.product.version: pc-i440fx-2.0 dmi.sys.vendor: QEMU To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1423672/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
