Public bug reported:
This bug has been introduced by an Oracle engineer, Chuck Lever, in
"net/sunrpc/svcsock.c" as a series of optimizations in commit
2b877fc53e975bdc5614c0a7d64047cfdbae3894 , on 2023-07-19. It breaks NFS-RPC
over TCP calls on clients under high load. In our case it is a bunch of rsync
daemons, syncing multiple mailboxes content.
As noted, we are using TCP protocol for NFS shares on Ubuntu 24.04, kernel 6.8,
and that's why we've run into that bug.
Here is the client DMESG log snippet:
=====================
[Fri Mar 21 05:26:02 2025] nfs: server ___________ not responding, still trying
[Fri Mar 21 05:26:02 2025] nfs: server ___________ not responding, still trying
[Fri Mar 21 05:26:02 2025] nfs: server ___________ not responding, still trying
[Fri Mar 21 05:26:02 2025] call_decode: 12 callbacks suppressed
[Fri Mar 21 05:26:02 2025] nfs: server ___________ OK
[Fri Mar 21 05:26:02 2025] nfs: server ___________ OK
[Fri Mar 21 05:26:02 2025] nfs: server ___________ OK
=====================
Fortunately Chuck Lever has rolled his patch back in
966a675da844f1a764bb44557c21561cc3d09840 on 2025-01-03.
Can we please cherry-pick this commit to kernel 6.8, since we've run into a
time-window when the fix is unavailable yet?
We are currently using Ubuntu 24.04 on both client and server, and the kernel
sources, cloned with "apt source linux" still have this commit un-reverted.
I have also un-patched these changes, and got my problem fixed, so sharing the
solution with you.
Thanks in advance!
ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: linux-image-6.8.0-55-generic 6.8.0-55.57
ProcVersionSignature: Ubuntu 6.8.0-55.57-generic 6.8.12
Uname: Linux 6.8.0-55-generic x86_64
NonfreeKernelModules: zfs
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Mar 21 11:55 seq
crw-rw---- 1 root audio 116, 33 Mar 21 11:55 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.28.1-0ubuntu3.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/timer',
'/dev/snd/seq'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: pass
Date: Fri Mar 21 12:06:45 2025
InstallationDate: Installed on 2025-02-13 (36 days ago)
InstallationMedia: Ubuntu-Server 24.04.1 LTS "Noble Numbat" - Release amd64
(20240827)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 1d6b:0107 Linux Foundation USB Virtual Hub
Bus 001 Device 003: ID 0557:9241 ATEN International Co., Ltd SMCI HID KM
Bus 001 Device 004: ID 0b1f:03ee Insyde Software Corp. RNDIS/Ethernet Gadget
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
MachineType: Supermicro Super Server
PciMultimedia:
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.8.0-55-generic
root=/dev/mapper/vg0-root ro
RelatedPackageVersions:
linux-restricted-modules-6.8.0-55-generic N/A
linux-backports-modules-6.8.0-55-generic N/A
linux-firmware 20240318.git3b128b60-0ubuntu2.10
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/23/2024
dmi.bios.release: 5.32
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: 2.3
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: X13SEB-TF
dmi.board.vendor: Supermicro
dmi.board.version: 1.01
dmi.chassis.asset.tag: Chassis Asset Tag
dmi.chassis.type: 17
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 0123456789
dmi.modalias:
dmi:bvnAmericanMegatrendsInternational,LLC.:bvr2.3:bd07/23/2024:br5.32:svnSupermicro:pnSuperServer:pvr0123456789:rvnSupermicro:rnX13SEB-TF:rvr1.01:cvnSupermicro:ct17:cvr0123456789:skuTobefilledbyO.E.M.:
dmi.product.family: Family
dmi.product.name: Super Server
dmi.product.sku: To be filled by O.E.M.
dmi.product.version: 0123456789
dmi.sys.vendor: Supermicro
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug noble
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2103802
Title:
NFS-over-TCP fix needs backported from master
Status in linux package in Ubuntu:
New
Bug description:
This bug has been introduced by an Oracle engineer, Chuck Lever, in
"net/sunrpc/svcsock.c" as a series of optimizations in commit
2b877fc53e975bdc5614c0a7d64047cfdbae3894 , on 2023-07-19. It breaks NFS-RPC
over TCP calls on clients under high load. In our case it is a bunch of rsync
daemons, syncing multiple mailboxes content.
As noted, we are using TCP protocol for NFS shares on Ubuntu 24.04, kernel
6.8, and that's why we've run into that bug.
Here is the client DMESG log snippet:
=====================
[Fri Mar 21 05:26:02 2025] nfs: server ___________ not responding, still
trying
[Fri Mar 21 05:26:02 2025] nfs: server ___________ not responding, still
trying
[Fri Mar 21 05:26:02 2025] nfs: server ___________ not responding, still
trying
[Fri Mar 21 05:26:02 2025] call_decode: 12 callbacks suppressed
[Fri Mar 21 05:26:02 2025] nfs: server ___________ OK
[Fri Mar 21 05:26:02 2025] nfs: server ___________ OK
[Fri Mar 21 05:26:02 2025] nfs: server ___________ OK
=====================
Fortunately Chuck Lever has rolled his patch back in
966a675da844f1a764bb44557c21561cc3d09840 on 2025-01-03.
Can we please cherry-pick this commit to kernel 6.8, since we've run into a
time-window when the fix is unavailable yet?
We are currently using Ubuntu 24.04 on both client and server, and the kernel
sources, cloned with "apt source linux" still have this commit un-reverted.
I have also un-patched these changes, and got my problem fixed, so sharing
the solution with you.
Thanks in advance!
ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: linux-image-6.8.0-55-generic 6.8.0-55.57
ProcVersionSignature: Ubuntu 6.8.0-55.57-generic 6.8.12
Uname: Linux 6.8.0-55-generic x86_64
NonfreeKernelModules: zfs
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Mar 21 11:55 seq
crw-rw---- 1 root audio 116, 33 Mar 21 11:55 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.28.1-0ubuntu3.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/timer',
'/dev/snd/seq'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: pass
Date: Fri Mar 21 12:06:45 2025
InstallationDate: Installed on 2025-02-13 (36 days ago)
InstallationMedia: Ubuntu-Server 24.04.1 LTS "Noble Numbat" - Release amd64
(20240827)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 1d6b:0107 Linux Foundation USB Virtual Hub
Bus 001 Device 003: ID 0557:9241 ATEN International Co., Ltd SMCI HID KM
Bus 001 Device 004: ID 0b1f:03ee Insyde Software Corp. RNDIS/Ethernet Gadget
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
MachineType: Supermicro Super Server
PciMultimedia:
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.8.0-55-generic
root=/dev/mapper/vg0-root ro
RelatedPackageVersions:
linux-restricted-modules-6.8.0-55-generic N/A
linux-backports-modules-6.8.0-55-generic N/A
linux-firmware 20240318.git3b128b60-0ubuntu2.10
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/23/2024
dmi.bios.release: 5.32
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: 2.3
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: X13SEB-TF
dmi.board.vendor: Supermicro
dmi.board.version: 1.01
dmi.chassis.asset.tag: Chassis Asset Tag
dmi.chassis.type: 17
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 0123456789
dmi.modalias:
dmi:bvnAmericanMegatrendsInternational,LLC.:bvr2.3:bd07/23/2024:br5.32:svnSupermicro:pnSuperServer:pvr0123456789:rvnSupermicro:rnX13SEB-TF:rvr1.01:cvnSupermicro:ct17:cvr0123456789:skuTobefilledbyO.E.M.:
dmi.product.family: Family
dmi.product.name: Super Server
dmi.product.sku: To be filled by O.E.M.
dmi.product.version: 0123456789
dmi.sys.vendor: Supermicro
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2103802/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
