Verification done for focal-proposed. No regressions in the consistent/flaky fail sets.
The flaky fail set had a long tail (tests that failed just a few times), but it can be isolated to another btrfs commit, and it doesn't seem to happen often. (e.g. not seen on focal-proposed, but seen on focal orig kernel on master-next, and both have the commit.) Thus not a general/permanent issue. More details in the next comment. Proposed kernel: --- $ uname -rv 4.15.0-125-generic #128-Ubuntu SMP Mon Nov 9 20:45:41 UTC 2020 $ VERSION='4.15.0-125-generic #128-Ubuntu' Number of runs: $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures: 14 $ RUNS=14 Consistent fail set: $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS 14 btrfs/186 14 btrfs/193 14 btrfs/200 14 btrfs/202 14 btrfs/204 14 btrfs/205 14 btrfs/214 14 btrfs/219 14 btrfs/220 14 btrfs/221 14 btrfs/222 14 generic/260 14 generic/286 14 generic/301 14 generic/371 14 generic/405 14 generic/407 14 generic/465 14 generic/480 14 generic/488 14 generic/502 14 generic/513 14 generic/526 14 generic/527 14 generic/546 14 generic/562 14 generic/565 Flaky fail set: $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS 1 btrfs/001 1 btrfs/147 1 btrfs/148 1 btrfs/149 1 btrfs/150 1 btrfs/151 1 btrfs/152 1 btrfs/153 1 btrfs/155 1 btrfs/156 1 btrfs/159 1 btrfs/164 1 btrfs/165 1 btrfs/166 1 btrfs/168 1 btrfs/169 1 btrfs/170 1 btrfs/171 1 btrfs/172 1 btrfs/173 1 btrfs/174 1 btrfs/177 1 btrfs/178 1 btrfs/179 1 btrfs/180 1 btrfs/181 1 btrfs/182 1 btrfs/183 1 btrfs/184 1 btrfs/187 1 btrfs/188 1 btrfs/190 1 btrfs/196 1 btrfs/197 1 btrfs/199 1 btrfs/206 1 btrfs/207 1 btrfs/208 1 btrfs/213 1 btrfs/217 1 generic/003 1 generic/017 1 generic/031 1 generic/033 1 generic/037 1 generic/038 1 generic/039 1 generic/040 1 generic/041 1 generic/043 1 generic/044 1 generic/045 1 generic/046 1 generic/047 1 generic/048 1 generic/049 1 generic/050 1 generic/051 1 generic/052 1 generic/053 1 generic/054 1 generic/055 1 generic/057 1 generic/059 1 generic/062 1 generic/064 1 generic/065 1 generic/066 1 generic/067 1 generic/068 1 generic/069 1 generic/071 1 generic/073 1 generic/076 1 generic/077 1 generic/079 1 generic/081 1 generic/082 1 generic/083 1 generic/084 1 generic/090 1 generic/103 1 generic/104 1 generic/297 1 generic/324 1 generic/325 1 generic/464 1 generic/518 2 btrfs/106 2 btrfs/107 2 btrfs/110 2 btrfs/111 2 btrfs/112 2 btrfs/114 2 btrfs/115 2 btrfs/116 2 btrfs/117 2 btrfs/118 2 btrfs/119 2 btrfs/120 2 btrfs/121 2 btrfs/122 2 btrfs/123 2 btrfs/126 2 btrfs/127 2 btrfs/128 2 btrfs/129 2 btrfs/130 2 btrfs/131 2 btrfs/132 2 btrfs/133 2 btrfs/134 2 btrfs/135 2 btrfs/136 2 btrfs/137 2 btrfs/138 2 btrfs/139 2 btrfs/144 2 btrfs/145 2 btrfs/146 2 btrfs/189 2 btrfs/191 2 generic/056 2 generic/085 2 generic/089 2 generic/101 2 generic/102 2 generic/509 2 generic/510 2 generic/601 2 generic/602 2 generic/603 2 generic/604 2 generic/605 2 generic/606 2 generic/607 2 generic/608 2 generic/610 2 generic/611 2 shared/002 2 shared/032 3 btrfs/076 3 btrfs/077 3 btrfs/079 3 btrfs/080 3 btrfs/082 3 btrfs/083 3 btrfs/084 3 btrfs/085 3 btrfs/086 3 btrfs/087 3 btrfs/088 3 btrfs/089 3 btrfs/090 3 btrfs/091 3 btrfs/092 3 btrfs/093 3 btrfs/094 3 btrfs/095 3 btrfs/096 3 btrfs/097 3 btrfs/098 3 btrfs/099 3 btrfs/102 3 btrfs/103 3 btrfs/104 3 btrfs/105 3 btrfs/108 3 btrfs/109 3 btrfs/113 3 btrfs/210 3 btrfs/211 3 btrfs/212 3 btrfs/215 3 btrfs/224 3 btrfs/226 3 generic/015 3 generic/018 3 generic/027 3 generic/029 3 generic/030 3 generic/032 3 generic/034 3 generic/600 4 btrfs/057 4 btrfs/058 4 btrfs/059 4 btrfs/060 4 btrfs/061 4 btrfs/062 4 btrfs/063 4 btrfs/064 4 btrfs/065 4 btrfs/066 4 btrfs/067 4 btrfs/068 4 btrfs/069 4 btrfs/070 4 btrfs/071 4 btrfs/072 4 btrfs/073 4 btrfs/074 4 btrfs/075 4 btrfs/081 4 btrfs/203 4 btrfs/209 4 generic/520 4 generic/589 4 generic/590 4 generic/592 4 generic/593 4 generic/594 4 generic/595 4 generic/599 5 btrfs/042 5 btrfs/043 5 btrfs/044 5 btrfs/045 5 btrfs/046 5 btrfs/047 5 btrfs/048 5 btrfs/049 5 btrfs/050 5 btrfs/051 5 btrfs/052 5 btrfs/053 5 btrfs/054 5 generic/476 5 generic/584 5 generic/585 5 generic/588 6 btrfs/035 6 btrfs/036 6 btrfs/037 6 btrfs/038 6 btrfs/039 6 btrfs/040 6 btrfs/041 6 btrfs/055 6 btrfs/056 6 generic/322 6 generic/573 6 generic/574 6 generic/575 6 generic/576 6 generic/577 6 generic/579 6 generic/580 6 generic/581 6 generic/582 6 generic/583 7 btrfs/026 7 btrfs/027 7 btrfs/029 7 btrfs/030 7 btrfs/031 7 btrfs/032 7 btrfs/033 7 btrfs/034 7 generic/531 7 generic/570 7 generic/572 8 btrfs/028 8 btrfs/201 8 generic/501 8 generic/563 8 generic/566 8 generic/567 8 generic/569 11 btrfs/078 11 generic/175 << increase, seen w/ test/patched kernel; from other btrfs patches. (comment #7) 13 btrfs/010 13 generic/074 13 generic/133 13 generic/142 13 generic/143 13 generic/475 ** Tags removed: verification-needed-bionic ** Tags added: verification-done-bionic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1902254 Title: Bionic: btrfs: kernel BUG at /build/linux- eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! Status in linux package in Ubuntu: Fix Committed Status in linux source package in Bionic: Fix Committed Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Fix Committed Bug description: [Impact] * Users of btrfs started hitting a kernel BUG() (below) after upgrade from 4.15.0-99.100 to 4.15.0-109.110, which has 55 btrfs changes. kernel BUG at /build/linux-eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! ... Krnl PSW : 00000000be9cb874 00000000ef3786e8 (btrfs_set_item_key_safe+0x152/0x1c0 [btrfs]) ... [...] Call Trace: [...] btrfs_set_item_key_safe+0x11c/0x1c0 [btrfs]) [...] __btrfs_drop_extents+0xb5a/0xda8 [btrfs] [...] btrfs_log_changed_extents+0x35c/0xaf0 [btrfs] [...] btrfs_log_inode+0x9ee/0x1080 [btrfs] [...] btrfs_log_inode_parent+0x224/0xa10 [btrfs] [...] btrfs_log_dentry_safe+0x80/0xa8 [btrfs] [...] btrfs_sync_file+0x392/0x550 [btrfs] [...] do_fsync+0x5e/0x90 [...] SyS_fdatasync+0x32/0x48 [...] system_call+0xd8/0x2c8 $ git log --oneline Ubuntu-4.15.0-99.100..Ubuntu-4.15.0-109.110 -- fs/btrfs/ | wc -l 55 * The error happens at random moments, regardless of a particular activity/load. Workaround is to downgrade. [Fix] * This BUG()/function is addressed in patch 4/4 [1] of series 'btrfs: Enhanced runtime defence against fuzzed images' [2], after issues in the real world, not just crafted fs images: 'one internal report has hit one BUG_ON() with real world fs' kernel BUG at fs/btrfs/ctree.c:3188! ... RIP: 0010:btrfs_set_item_key_safe+0x16c/0x180 * The patch/set [3] is applied in v5.10-rc1 and Ubuntu Unstable: - d16c702fe4f2 btrfs: ctree: check key order before merging tree blocks - 07cce5cf3b48 btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref() - 1c2a07f598d5 btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent() - f98b6215d7d1 btrfs: extent_io: do extra check for extent buffer read write functions [Test Case] * There is working synthetic reproducer for this issue, which is hard to reproduce as reported in commit [4] that introduces debugging for the issue. * Regression tests with xfstests and stress-ng shows no regressions between un/patched kernels. [Other Info] * Trivial backports (only refreshing a few context lines) with 3 more dependency patches on Bionic and 1 on Focal. And Bionic needed one extra hunk to '#include' a header. Groovy all apply cleanly. [1] https://lore.kernel.org/linux-btrfs/20200819063550.62832-5-...@suse.com/ [2] https://lore.kernel.org/linux-btrfs/20200819063550.62832-1-...@suse.com/ [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d16c702fe4f274bd77b47d3ab737eadcf24e0b93 [4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7c15d41016dc886cc011e3854d855e219759ae68 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1902254/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp