Andrea, thanks. What about other fs/fuse/ patches marked as @stable? We have more problems reported on 4.15.x Ubuntu kernel (see e.g. https://github.com/hanwen/go-fuse/issues/287) and that is unfortunate when the fix/patch is known, marked as needed to be applied to stable kernel, but Ubuntu is not picking those patches up, just because upstream 4.15 kernel is not LTS release and stopped being updated long ago, unlike e.g. upstream 4.4 and 4.14 stable releases which are updated to this day and have all those stable patches I'm talking about.
** Bug watch added: github.com/hanwen/go-fuse/issues #287 https://github.com/hanwen/go-fuse/issues/287 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1823972 Title: bionic, xenial/hwe: misses "fuse: fix initial parallel dirops" patch Status in linux package in Ubuntu: Confirmed Bug description: SRU Justification: [Impact] * Enabling parallel dirops in fuse (FUSE_PARALLEL_DIROPS) may cause a race condition that leave fuse inode's mutex held, triggering a deadlock * The problem is that the lock and unlock paths are relying on get_fuse_conn(inode)->parallel_dirops to decide if the mutex needs to be acquired/released, but its value might be set in the lock path and unset in the unlock path (leaving the mutex held) [Test Case] * A test case that triggers the bug almost immediately can be found here https://github.com/hanwen/go-fuse/pull/288 [Fix] * Instead of relying on get_fuse_conn(inode)->parallel_dirops both in fuse_lock_inode() and fuse_unlock_inode(), only check this flag in the locking path and pass a variable to fuse_unlock_inode() to determine if the mutex was acquired or not [Regression Potential] * Fix has been tested on the affected platform. It is an upstream fix that seems to affect only 4.7+ kernels, more exactly in our case only Bionic kernels (and derived) are affected. Cosmic and above already include this fix. So regression potential is minimal. [Original bug report] Hello up there, We were reported about a deadlock in the kernel while using a FUSE-based filesystem on Ubuntu. The kernel in question is Ubuntu-hwe-4.15.0-47.50~16.04.1 from Xenial/HWE. We tracked this bug to the fact that 4.15.x kernel in Ubuntu does not include the following patch, in despite the patch being marked as needed for v4.7+ stable kernels: https://git.kernel.org/linus/63576c13bd Please see the following go-fuse issue for full details: https://github.com/hanwen/go-fuse/issues/281#issuecomment-480008562 The bug is potentially applicable to libfuse users too since libfuse by default enables parallel dirops whenever kernel claims support for it, which libfuse maintained confirmed: https://github.com/hanwen/go-fuse/issues/281#issuecomment-480013202 https://github.com/hanwen/go-fuse/issues/281#issuecomment-480510381 We tested that cherry-picking 63576c13bd into 4.15.x series makes the problem go away: https://github.com/hanwen/go-fuse/issues/281#issuecomment-480499969 So please include https://git.kernel.org/linus/63576c13bd into Ubuntu 4.15.x kernel series which are bionic/master and xenial/hwe, and which currently don't have this patch. -------- Here is a full list of FUSE patches marked to be needed in stable kernels starting from v4.15: kirr@deco:~/src/linux/linux$ git log --oneline v4.15..v5.1-rc3 --grep="stable@" -- fs/fuse/ a2ebba824106 fuse: decrement NR_WRITEBACK_TEMP on the right page 9509941e9c53 fuse: call pipe_buf_release() under pipe lock 8a3177db59cd cuse: fix ioctl 97e1532ef81a fuse: handle zero sized retrieve correctly 2e64ff154ce6 fuse: continue to send FUSE_RELEASEDIR when FUSE_OPEN returns ENOSYS ebacb8127359 fuse: fix use-after-free in fuse_direct_IO() 2d84a2d19b61 fuse: fix possibly missed wake-up after abort 7fabaf303458 fuse: fix leaked notify reply 908a572b80f6 fuse: fix blocked_waitq wakeup 4c316f2f3ff3 fuse: set FR_SENT while locked d2d2d4fb1f54 fuse: Fix use-after-free in fuse_dev_do_write() bc78abbd55dd fuse: Fix use-after-free in fuse_dev_do_read() a2477b0e67c5 fuse: Don't access pipe->buffers without pipe_lock() 63576c13bd17 fuse: fix initial parallel dirops e8f3bd773d22 fuse: Fix oops at process_init_reply() b8f95e5d13f5 fuse: umount should wait for all requests 45ff350bbd9d fuse: fix unlocked access to processing queue 87114373ea50 fuse: fix double request_end() 543b8f8662fe (tag: fuse-update-4.18) fuse: don't keep dead fuse_conn at fuse_fill_super(). 6becdb601bae fuse: fix control dir setup and teardown 8a301eb16d99 fuse: fix congested state leak on aborted connections df0e91d48827 fuse: atomic_o_trunc should truncate pagecache Among those only 8a3177db59cd and 2d84a2d19b61 should not be applied to 4.15.x becuase they cure a problem introduced in a later kernel (please see got log without --oneline for stable@ details) However both bionic and xenial/hwe has much less fuse patches applied: kirr@deco:~/src/linux/linux$ git log --oneline v4.15..bionic/master -- fs/fuse/ e992e3521885 fuse: fix control dir setup and teardown f3a3e0537dcd fuse: don't keep dead fuse_conn at fuse_fill_super(). 840c77082f93 fuse: atomic_o_trunc should truncate pagecache c0e31b214498 fuse: fix congested state leak on aborted connections 45f23c59120f UBUNTU: SAUCE: (namespace) fuse: Allow user namespace mounts 1223588451c6 UBUNTU: SAUCE: (namespace) fuse: Restrict allow_other to the superblock's namespace or a descendant b4d1889491a0 UBUNTU: SAUCE: (namespace) fuse: Support fuse filesystems outside of init_user_ns kirr@deco:~/src/linux/linux$ git log --oneline v4.15..xenial/hwe -- fs/fuse/ e992e3521885 fuse: fix control dir setup and teardown f3a3e0537dcd fuse: don't keep dead fuse_conn at fuse_fill_super(). 840c77082f93 fuse: atomic_o_trunc should truncate pagecache c0e31b214498 fuse: fix congested state leak on aborted connections 45f23c59120f UBUNTU: SAUCE: (namespace) fuse: Allow user namespace mounts 1223588451c6 UBUNTU: SAUCE: (namespace) fuse: Restrict allow_other to the superblock's namespace or a descendant b4d1889491a0 UBUNTU: SAUCE: (namespace) fuse: Support fuse filesystems outside of init_user_ns which suggests that other FUSE fixes should be cherry-picked too. Please consider cherry-picking those additional patches too. They are all in upstream kernel stable series, e.g. stable/linux-4.14.y has them: kirr@deco:~/src/linux/linux$ git log --oneline v4.14..stable/linux-4.14.y -- fs/fuse/ 266a69895b89 fuse: handle zero sized retrieve correctly b928e93d864c fuse: decrement NR_WRITEBACK_TEMP on the right page 65f222bb370e fuse: call pipe_buf_release() under pipe lock c1149b873482 fuse: continue to send FUSE_RELEASEDIR when FUSE_OPEN returns ENOSYS 6ceec07cc84a fuse: fix leaked notify reply a42d933dc281 fuse: fix use-after-free in fuse_direct_IO() 78da72ee42d8 fuse: set FR_SENT while locked f6f21a2b70c6 fuse: fix blocked_waitq wakeup ab962e91008a fuse: Fix use-after-free in fuse_dev_do_write() d94b3a2375cb fuse: Fix use-after-free in fuse_dev_do_read() e8a3f3a03655 fuse: Add missed unlock_page() to fuse_readpages_fill() ff4a71855d0a fuse: Fix oops at process_init_reply() 973206923812 fuse: umount should wait for all requests fc17d7519e8e fuse: fix unlocked access to processing queue cfb6eca6e4bb fuse: fix double request_end() 7d392674443c fuse: fix initial parallel dirops eaebcf902ae0 fuse: Don't access pipe->buffers without pipe_lock() 69829f749a43 fuse: fix control dir setup and teardown 3a37d85a90da fuse: don't keep dead fuse_conn at fuse_fill_super(). 2f7bf369b5f8 fuse: atomic_o_trunc should truncate pagecache 02832578eb9d fuse: fix congested state leak on aborted connections and it is just that stable/linux-4.15.y stopped being maintained by Greg KH. Thanks beforehand, Kirill --- ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CurrentDesktop: Unity DistroRelease: Ubuntu 16.04 HibernationDevice: RESUME=UUID=dec25862-42c7-4494-8f75-e6cc76aa65ea InstallationDate: Installed on 2019-02-28 (39 days ago) InstallationMedia: Ubuntu 16.04.6 LTS "Xenial Xerus" - Release amd64 (20190227) IwConfig: lo no wireless extensions. ens3 no wireless extensions. Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: QEMU Standard PC (i440FX + PIIX, 1996) Package: linux (not installed) ProcFB: 0 virtiodrmfb ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.15.0-45-generic root=UUID=fe1f50d9-7142-4ee3-8904-bae7d80bc6b1 ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.15.0-45.48~16.04.1-generic 4.15.18 RelatedPackageVersions: linux-restricted-modules-4.15.0-45-generic N/A linux-backports-modules-4.15.0-45-generic N/A linux-firmware 1.157.21 RfKill: Tags: xenial Uname: Linux 4.15.0-45-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo _MarkForUpload: True dmi.bios.date: 04/01/2014 dmi.bios.vendor: SeaBIOS dmi.bios.version: 1.12.0-1 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-i440fx-3.1 dmi.modalias: dmi:bvnSeaBIOS:bvr1.12.0-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-3.1:cvnQEMU:ct1:cvrpc-i440fx-3.1: dmi.product.name: Standard PC (i440FX + PIIX, 1996) dmi.product.version: pc-i440fx-3.1 dmi.sys.vendor: QEMU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1823972/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
