apport information
** Tags added: apport-collected noble
** Description changed:
Steps to reproduce :
1. Power on the NVMeoF enclosure.
2. Discover and connect the drives.
3. Create 2 zpools with even and odd drives.
4. Start the IO on both pools created.
Observation :
1. Observed call trace while running ZFS IO Able to see "failed to send
request-5" and drive went continuously reconnected state.
2. The issue is seen with Ubuntu 24.04.1 with kernel 6.8.0-49.generic kernel.
From Kernel ring buffer logs (dmesg) :
[Tue Feb 11 05:25:55 2025] ------------[ cut here ]------------
[Tue Feb 11 05:25:55 2025] WARNING: CPU: 10 PID: 114873 at
net/core/skbuff.c:7006 skb_splice_from_iter+0x139/0x370
[Tue Feb 11 05:25:55 2025] Modules linked in: nvme_tcp nvme_keyring nvme
xt_tcpudp nft_compat nf_tables qrtr cfg80211 binfmt_misc zfs(PO) spl(O)
intel_rapl_msr intel_rapl_common intel_uncore_frequency
intel_uncore_frequency_common sb_edac x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel dell_wmi dell_smbios dell_wmi_descriptor kvm video mgag200
ledtrig_audio irqbypass sparse_keymap dcdbas joydev input_leds mei_me
i2c_algo_bit mei acpi_power_meter rapl intel_cstate lpc_ich ipmi_ssif mac_hid
acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler mxm_wmi sch_fq_codel
dm_multipath nvme_fabrics msr nvme_core nvme_auth efi_pstore nfnetlink
dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456
async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
libcrc32c raid1 raid0 mlx5_ib ib_uverbs macsec ib_core mlx5_core
crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic
ghash_clmulni_intel sha256_ssse3 sha1_ssse3 mlxfw psample tls pci_hyperv_intf
tg3 pata_acpi wmi hid_generic usbhid hid aesni_intel
[Tue Feb 11 05:25:55 2025] crypto_simd cryptd
[Tue Feb 11 05:25:55 2025] CPU: 10 PID: 114873 Comm: kworker/10:2H Tainted: P
O 6.8.0-49-generic #49-Ubuntu
[Tue Feb 11 05:25:55 2025] Hardware name: Dell Inc. PowerEdge R730/072T6D,
BIOS 2.7.1 001/22/2018
[Tue Feb 11 05:25:55 2025] Workqueue: nvme_tcp_wq nvme_tcp_io_work [nvme_tcp]
[Tue Feb 11 05:25:55 2025] RIP: 0010:skb_splice_from_iter+0x139/0x370
[Tue Feb 11 05:25:55 2025] Code: 39 e1 48 8b 53 08 49 0f 47 cc 49 89 cd f6 c2
01 0f 85 c0 01 00 00 66 90 48 89 da 48 8b 12 80 e6 08 0f 84 8e 00 00 00 4d 89
fe <0f> 0b 49 c7 c0 fb ff ff ff 48 8b 85 68 ff ff ff 41 01 46 70 41 01
[Tue Feb 11 05:25:55 2025] RSP: 0018:ffffb216769d7a38 EFLAGS: 00010202
[Tue Feb 11 05:25:55 2025] RAX: 0000000000000000 RBX: fffff74820347000 RCX:
0000000000001000
[Tue Feb 11 05:25:55 2025] RDX: 0017ffffc0000840 RSI: 0000000000000000 RDI:
0000000000000000
[Tue Feb 11 05:25:55 2025] RBP: ffffb216769d7ae0 R08: 0000000000000000 R09:
0000000000000000
[Tue Feb 11 05:25:55 2025] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000001000
[Tue Feb 11 05:25:55 2025] R13: 0000000000001000 R14: ffff9c22fccbfe00 R15:
ffff9c22fccbfe00
[Tue Feb 11 05:25:55 2025] FS: 0000000000000000(0000)
GS:ffff9c347f680000(0000) knlGS:0000000000000000
[Tue Feb 11 05:25:55 2025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Tue Feb 11 05:25:55 2025] CR2: 00007d79ae7af000 CR3: 0000002a226e4001 CR4:
00000000003706f0
[Tue Feb 11 05:25:55 2025] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[Tue Feb 11 05:25:55 2025] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[Tue Feb 11 05:25:55 2025] Call Trace:
[Tue Feb 11 05:25:55 2025] <TASK>
[Tue Feb 11 05:25:55 2025] ? show_regs+0x6d/0x80
[Tue Feb 11 05:25:55 2025] ? __warn+0x89/0x160
[Tue Feb 11 05:25:55 2025] ? skb_splice_from_iter+0x139/0x370
[Tue Feb 11 05:25:55 2025] ? report_bug+0x17e/0x1b0
[Tue Feb 11 05:25:55 2025] ? handle_bug+0x51/0xa0
[Tue Feb 11 05:25:55 2025] ? exc_invalid_op+0x18/0x80
[Tue Feb 11 05:25:55 2025] ? asm_exc_invalid_op+0x1b/0x20
[Tue Feb 11 05:25:55 2025] ? skb_splice_from_iter+0x139/0x370
[Tue Feb 11 05:25:55 2025] ? skb_splice_from_iter+0xd5/0x370
[Tue Feb 11 05:25:55 2025] tcp_sendmsg_locked+0x352/0xd70
[Tue Feb 11 05:25:55 2025] ? tcp_push+0x159/0x190
[Tue Feb 11 05:25:55 2025] ? tcp_sendmsg_locked+0x9c4/0xd70
[Tue Feb 11 05:25:55 2025] tcp_sendmsg+0x2c/0x50
[Tue Feb 11 05:25:55 2025] inet_sendmsg+0x42/0x80
[Tue Feb 11 05:25:55 2025] sock_sendmsg+0x118/0x150
[Tue Feb 11 05:25:55 2025] nvme_tcp_try_send_data+0x16e/0x4d0 [nvme_tcp]
[Tue Feb 11 05:25:55 2025] nvme_tcp_try_send+0x23c/0x300 [nvme_tcp]
[Tue Feb 11 05:25:55 2025] nvme_tcp_io_work+0x40/0xe0 [nvme_tcp]
[Tue Feb 11 05:25:55 2025] process_one_work+0x178/0x350
[Tue Feb 11 05:25:55 2025] worker_thread+0x306/0x440
[Tue Feb 11 05:25:55 2025] ? __pfx_worker_thread+0x10/0x10
[Tue Feb 11 05:25:55 2025] kthread+0xf2/0x120
[Tue Feb 11 05:25:55 2025] ? __pfx_kthread+0x10/0x10
[Tue Feb 11 05:25:55 2025] ret_from_fork+0x47/0x70
[Tue Feb 11 05:25:55 2025] ? __pfx_kthread+0x10/0x10
[Tue Feb 11 05:25:55 2025] ret_from_fork_asm+0x1b/0x30
[Tue Feb 11 05:25:55 2025] </TASK>
[Tue Feb 11 05:25:55 2025] ---[ end trace 0000000000000000 ]---
[Tue Feb 11 05:25:55 2025] nvme nvme8: failed to send request -5
[Tue Feb 11 05:26:25 2025] nvme nvme8: I/O tag 5 (9005) type 4 opcode 0x2
(I/O Cmd) QID 11 timeout
[Tue Feb 11 05:26:25 2025] nvme nvme8: starting error recovery
[Tue Feb 11 05:26:25 2025] nvme nvme8: I/O tag 6 (c006) type 4 opcode 0x1
(I/O Cmd) QID 11 timeout
[Tue Feb 11 05:26:25 2025] nvme nvme8: I/O tag 7 (d007) type 4 opcode 0x2
(I/O Cmd) QID 11 timeout
[Tue Feb 11 05:26:25 2025] nvme nvme8: I/O tag 11 (700b) type 4 opcode 0x1
(I/O Cmd) QID 11 timeout
[Tue Feb 11 05:26:25 2025] nvme nvme8: I/O tag 12 (300c) type 4 opcode 0x1
(I/O Cmd) QID 11 timeout
[Tue Feb 11 05:26:25 2025] nvme nvme32: failed to send request -5
[Tue Feb 11 05:26:25 2025] nvme nvme8: Reconnecting in 10 seconds...
[Tue Feb 11 05:26:25 2025] nvme nvme32: starting error recovery
[Tue Feb 11 05:26:25 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:25 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:25 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:25 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:25 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:25 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:25 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:25 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:25 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:25 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:25 2025] nvme nvme32: Reconnecting in 10 seconds...
[Tue Feb 11 05:26:36 2025] nvme nvme8: queue_size 128 > ctrl sqsize 16,
clamping down
[Tue Feb 11 05:26:36 2025] nvme nvme8: creating 16 I/O queues.
[Tue Feb 11 05:26:36 2025] nvme nvme32: queue_size 128 > ctrl sqsize 16,
clamping down
[Tue Feb 11 05:26:36 2025] nvme nvme32: creating 16 I/O queues.
[Tue Feb 11 05:26:36 2025] nvme nvme8: mapped 16/0/0 default/read/poll queues.
[Tue Feb 11 05:26:36 2025] nvme nvme8: Successfully reconnected (1 attempt)
[Tue Feb 11 05:26:36 2025] nvme nvme8: failed to send request -5
[Tue Feb 11 05:26:36 2025] nvme nvme32: mapped 16/0/0 default/read/poll
queues.
[Tue Feb 11 05:26:36 2025] nvme nvme8: starting error recovery
[Tue Feb 11 05:26:36 2025] nvme_ns_head_submit_bio: 55 callbacks suppressed
[Tue Feb 11 05:26:36 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:36 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:36 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:36 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:36 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:36 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:36 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:36 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:36 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:36 2025] block nvme8n1: no usable path - requeuing I/O
[Tue Feb 11 05:26:36 2025] nvme nvme32: Successfully reconnected (1 attempt)
[Tue Feb 11 05:26:36 2025] nvme nvme8: reading non-mdts-limits failed: -4
+ ---
+ ProblemType: Bug
+ AlsaDevices:
+ total 0
+ crw-rw---- 1 root audio 116, 1 Feb 12 07:46 seq
+ crw-rw---- 1 root audio 116, 33 Feb 12 07:46 timer
+ AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
+ ApportVersion: 2.28.1-0ubuntu2
+ Architecture: amd64
+ ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
+ AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/timer',
'/dev/snd/seq'] failed with exit code 1:
+ CRDA: N/A
+ CasperMD5CheckResult: unknown
+ DistroRelease: Ubuntu 24.04
+ InstallationDate: Installed on 2024-09-20 (146 days ago)
+ InstallationMedia: Ubuntu-Server 24.04 LTS "Noble Numbat" - Release amd64
(20240423)
+ IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
+ MachineType: Dell Inc. PowerEdge R740
+ NonfreeKernelModules: zfs
+ Package: linux (not installed)
+ PciMultimedia:
+
+ ProcEnviron:
+ LANG=en_US.UTF-8
+ PATH=(custom, no user)
+ SHELL=/bin/bash
+ TERM=xterm-256color
+ XDG_RUNTIME_DIR=<set>
+ ProcFB: 0 mgag200drmfb
+ ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.8.0-52-generic
root=/dev/mapper/ubuntu--vg-ubuntu--lv ro intel_iommu=off
+ ProcVersionSignature: Ubuntu 6.8.0-52.53-generic 6.8.12
+ RelatedPackageVersions:
+ linux-restricted-modules-6.8.0-52-generic N/A
+ linux-backports-modules-6.8.0-52-generic N/A
+ linux-firmware 20240318.git3b128b60-0ubuntu2.3
+ RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
+ Tags: noble
+ Uname: Linux 6.8.0-52-generic x86_64
+ UpgradeStatus: No upgrade log present (probably fresh install)
+ UserGroups: N/A
+ _MarkForUpload: True
+ dmi.bios.date: 06/04/2023
+ dmi.bios.release: 2.19
+ dmi.bios.vendor: Dell Inc.
+ dmi.bios.version: 2.19.1
+ dmi.board.name: 08D89F
+ dmi.board.vendor: Dell Inc.
+ dmi.board.version: A03
+ dmi.chassis.type: 23
+ dmi.chassis.vendor: Dell Inc.
+ dmi.modalias:
dmi:bvnDellInc.:bvr2.19.1:bd06/04/2023:br2.19:svnDellInc.:pnPowerEdgeR740:pvr:rvnDellInc.:rn08D89F:rvrA03:cvnDellInc.:ct23:cvr:skuSKU=0715;ModelName=PowerEdgeR740:
+ dmi.product.family: PowerEdge
+ dmi.product.name: PowerEdge R740
+ dmi.product.sku: SKU=0715;ModelName=PowerEdge R740
+ dmi.sys.vendor: Dell Inc.
** Attachment added: "CurrentDmesg.txt"
https://bugs.launchpad.net/bugs/2098056/+attachment/5857059/+files/CurrentDmesg.txt
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2098056
Title:
RAID getting corrupted while running ZFS IOs .
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2098056/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
