Public bug reported:
I've 3 ceph (osd, mon at all, and mds on two of them) servers.
CephFS (with quota enabled) pool mounted on this servers to /home
root@regio:~# uname -a
Linux regio 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024
x86_64 x86_64 x86_64 GNU/Linux
ceph installed by cephadm (curl method) and has version 18.2.4 reef
(stable)
/etc/fstab on problem servers (on this 3 ceph servers) :
home_mana...@f0501afe-4d26-11ef-a71d-b370fff6f286.data=/volumes/_nogroup/home/28dd2985-d00e-4157-b116-bd5f626136b6
/home ceph
mon_addr=tribus/regio/phylon,secretfile=/etc/ceph/ceph.client.home_manager.keyring,noatime,_netdev
0 0
Some applications (on every ceph server) long time writes to /home/$USER
and in random moment the /home becomes not available with error:
root@regio:~# ll /home
ls: cannot access '/home': Permission denied
Workaround is:
umount -l /home && mount /home
dmesg log flooded by:
[Sun Aug 25 23:23:22 2024] ceph: check_quota_exceeded: ino
(100016a4608.fffffffffffffffe) null i_snap_realm
[Sun Aug 25 23:23:22 2024] ceph: check_quota_exceeded: ino
(100016a4608.fffffffffffffffe) null i_snap_realm
[Sun Aug 25 23:23:22 2024] ceph: check_quota_exceeded: ino
(100016a4608.fffffffffffffffe) null i_snap_realm
[Sun Aug 25 23:23:22 2024] ceph: check_quota_exceeded: ino
(100016a4608.fffffffffffffffe) null i_snap_realm
[Sun Aug 25 23:23:27 2024] check_quota_exceeded: 939 callbacks suppressed
[Sun Aug 25 23:23:27 2024] ceph: check_quota_exceeded: ino
(100016a4608.fffffffffffffffe) null i_snap_realm
[Sun Aug 25 23:23:27 2024] ceph: check_quota_exceeded: ino
(100016a4608.fffffffffffffffe) null i_snap_realm
....
and some useful dmesg logs:
dmesg -T | egrep -v 'check_quota_exceeded|get_quota_realm'
[Mon Aug 26 00:34:03 2024] libceph: mon2 (1)192.168.30.20:6789 session lost,
hunting for new mon
[Mon Aug 26 00:34:03 2024] libceph: mon1 (1)192.168.30.21:6789 session
established
[Mon Aug 26 05:40:59 2024] ceph: mds0 caps renewed
[Mon Aug 26 10:00:44 2024] process 'Programs/ncbi-blast-2.2.31+/bin/blastp'
started with executable stack
[Mon Aug 26 13:07:40 2024] libceph: mon0 (1)192.168.30.20:6789 session lost,
hunting for new mon
[Mon Aug 26 13:07:40 2024] libceph: mon2 (1)192.168.30.22:6789 session
established
[Mon Aug 26 13:08:29 2024] ceph: mds0 caps stale
[Mon Aug 26 13:08:29 2024] ceph: mds1 caps stale
[Mon Aug 26 13:08:29 2024] ceph: mds1 caps renewed
[Mon Aug 26 13:08:29 2024] ceph: mds0 caps went stale, renewing
[Mon Aug 26 13:08:29 2024] ceph: mds0 caps stale
[Mon Aug 26 13:08:29 2024] ceph: mds0 caps renewed
[Mon Aug 26 13:08:30 2024] libceph: mds1 (1)192.168.30.20:6801 socket closed
(con state OPEN)
[Mon Aug 26 13:08:30 2024] libceph: mds0 (1)192.168.30.22:6801 socket closed
(con state OPEN)
[Mon Aug 26 13:08:30 2024] libceph: mds0 (1)192.168.30.22:6801 session reset
[Mon Aug 26 13:08:30 2024] ceph: mds0 closed our session
[Mon Aug 26 13:08:30 2024] ceph: mds0 reconnect start
[Mon Aug 26 13:08:30 2024] ceph: mds0 reconnect denied
[Mon Aug 26 13:08:30 2024] remove_session_caps_cb: 1030124 callbacks suppressed
[Mon Aug 26 13:08:30 2024] ceph: dropping dirty Fw state for 0000000026032bb4
1099573918106
[Mon Aug 26 13:08:30 2024] ceph: dropping dirty Fw state for 00000000f4a0b9f9
1099577329230
[Mon Aug 26 13:08:30 2024] ceph: dropping dirty Fw state for 00000000dc884e74
1099573918192
[Mon Aug 26 13:08:30 2024] ceph: dropping dirty Fw state for 0000000044945324
1099562341483
[Mon Aug 26 13:08:30 2024] ceph: dropping dirty Fw state for 00000000af07ec63
1099577329203
[Mon Aug 26 13:08:30 2024] ceph: dropping dirty Fw state for 00000000641be027
1099577329196
[Mon Aug 26 13:08:30 2024] libceph: mds0 (1)192.168.30.22:6801 socket closed
(con state V1_CONNECT_MSG)
[Mon Aug 26 13:08:30 2024] libceph: mds1 (1)192.168.30.20:6801 session reset
[Mon Aug 26 13:08:30 2024] ceph: mds1 closed our session
[Mon Aug 26 13:08:30 2024] ceph: mds1 reconnect start
[Mon Aug 26 13:08:30 2024] ceph: mds1 reconnect denied
[Mon Aug 26 13:08:30 2024] ceph: mds0 rejected session
[Mon Aug 26 13:08:30 2024] libceph: mds1 (1)192.168.30.20:6801 socket closed
(con state V1_CONNECT_MSG)
[Mon Aug 26 13:08:31 2024] ceph: mds1 rejected session
[Mon Aug 26 13:38:30 2024] libceph: mds0 (1)192.168.30.22:6801 socket closed
(con state OPEN)
[Mon Aug 26 13:38:31 2024] libceph: mds1 (1)192.168.30.20:6801 socket closed
(con state OPEN)
[Mon Aug 26 14:38:12 2024] ceph: No path or : separator in source
[Mon Aug 26 14:38:12 2024] libceph: mon2 (1)192.168.30.20:6789 session
established
[Mon Aug 26 14:38:12 2024] libceph: client147835 fsid
f0501afe-4d26-11ef-a71d-b370fff6f286
[Mon Aug 26 15:42:30 2024] libceph: osd3 down
[Mon Aug 26 15:42:31 2024] libceph: osd2 down
[Mon Aug 26 15:42:31 2024] libceph: osd8 down
[Mon Aug 26 15:42:36 2024] libceph: mon2 (1)192.168.30.22:6789 session lost,
hunting for new mon
[Mon Aug 26 15:42:36 2024] libceph: mon0 (1)192.168.30.20:6789 session
established
[Mon Aug 26 15:42:58 2024] ceph: mds0 caps stale
[Mon Aug 26 15:42:58 2024] ceph: mds0 caps stale
[Mon Aug 26 15:43:24 2024] ceph: mds0 reconnect start
[Mon Aug 26 15:43:24 2024] ceph: mds0 reconnect success
[Mon Aug 26 15:43:25 2024] ceph: mds0 reconnect start
[Mon Aug 26 15:43:25 2024] libceph: mon2 (1)192.168.30.20:6789 session lost,
hunting for new mon
[Mon Aug 26 15:43:25 2024] ceph: mds0 reconnect success
[Mon Aug 26 15:43:25 2024] libceph: mon0 (1)192.168.30.20:6789 session
established
[Mon Aug 26 15:44:14 2024] ceph: mds0 recovery completed
[Mon Aug 26 15:44:14 2024] ceph: mds0 recovery completed
[Mon Aug 26 15:44:15 2024] libceph: osd8 up
[Mon Aug 26 15:44:29 2024] libceph: osd3 up
[Mon Aug 26 15:44:34 2024] libceph: osd3 down
[Mon Aug 26 15:44:34 2024] libceph: osd2 down
[Mon Aug 26 15:44:34 2024] libceph: osd8 down
[Mon Aug 26 15:44:34 2024] libceph: osd8 up
[Mon Aug 26 15:44:34 2024] libceph: osd3 up
[Mon Aug 26 15:44:39 2024] libceph: osd2 up
[Mon Aug 26 15:45:17 2024] libceph: mon0 (1)192.168.30.20:6789 socket closed
(con state OPEN)
[Mon Aug 26 15:45:17 2024] libceph: mon0 (1)192.168.30.20:6789 session lost,
hunting for new mon
[Mon Aug 26 15:45:17 2024] libceph: mon0 (1)192.168.30.20:6789 socket closed
(con state OPEN)
[Mon Aug 26 15:45:17 2024] libceph: mon0 (1)192.168.30.20:6789 session lost,
hunting for new mon
[Mon Aug 26 15:45:17 2024] libceph: mon2 (1)192.168.30.22:6789 session
established
[Mon Aug 26 15:45:17 2024] libceph: mon1 (1)192.168.30.21:6789 session
established
[Mon Aug 26 15:48:08 2024] libceph: mon2 (1)192.168.30.22:6789 socket closed
(con state OPEN)
[Mon Aug 26 15:48:08 2024] libceph: mon2 (1)192.168.30.22:6789 session lost,
hunting for new mon
[Mon Aug 26 15:48:08 2024] libceph: mon0 (1)192.168.30.20:6789 session
established
[Mon Aug 26 15:50:12 2024] libceph: osd8 down
[Mon Aug 26 15:50:17 2024] libceph: osd2 down
[Mon Aug 26 15:50:17 2024] libceph: osd3 down
[Mon Aug 26 15:52:16 2024] libceph: osd2 up
[Mon Aug 26 15:52:16 2024] libceph: osd3 up
[Mon Aug 26 15:52:16 2024] libceph: osd8 up
[Mon Aug 26 16:25:35 2024] ixgbe 0000:62:00.1: removed PHC on enp98s0f1
[Mon Aug 26 16:25:36 2024] ixgbe 0000:62:00.0: removed PHC on enp98s0f0
[Mon Aug 26 16:25:36 2024] bond0: (slave enp98s0f1): link status definitely
down, disabling slave
[Mon Aug 26 16:25:36 2024] bond0: (slave enp98s0f0): link status definitely
down, disabling slave
[Mon Aug 26 16:25:36 2024] bond0: now running without any active interface!
[Mon Aug 26 16:25:37 2024] pps pps0: new PPS source ptp2
[Mon Aug 26 16:25:37 2024] ixgbe 0000:62:00.1: registered PHC device on
enp98s0f1
[Mon Aug 26 16:25:37 2024] 8021q: adding VLAN 0 to HW filter on device enp98s0f1
[Mon Aug 26 16:25:37 2024] pps pps1: new PPS source ptp3
[Mon Aug 26 16:25:37 2024] ixgbe 0000:62:00.0: registered PHC device on
enp98s0f0
[Mon Aug 26 16:25:38 2024] 8021q: adding VLAN 0 to HW filter on device enp98s0f0
[Mon Aug 26 16:25:38 2024] bond0: (slave enp98s0f1): link status definitely
down, disabling slave
[Mon Aug 26 16:25:38 2024] bond0: (slave enp98s0f0): link status definitely
down, disabling slave
[Mon Aug 26 16:25:42 2024] ixgbe 0000:62:00.1 enp98s0f1: NIC Link is Up 10
Gbps, Flow Control: None
[Mon Aug 26 16:25:42 2024] bond0: (slave enp98s0f1): link status definitely up,
10000 Mbps full duplex
[Mon Aug 26 16:25:42 2024] bond0: active interface up!
[Mon Aug 26 16:25:59 2024] ixgbe 0000:62:00.0 enp98s0f0: NIC Link is Up 5 Gbps,
Flow Control: None
[Mon Aug 26 16:25:59 2024] bond0: (slave enp98s0f0): link status definitely up,
5000 Mbps full duplex
[Mon Aug 26 16:51:25 2024] libceph: osd8 down
[Mon Aug 26 16:51:25 2024] libceph: osd2 down
[Mon Aug 26 16:51:25 2024] libceph: osd3 down
[Mon Aug 26 16:51:25 2024] libceph: osd2 up
[Mon Aug 26 16:51:25 2024] libceph: osd3 up
[Mon Aug 26 16:51:25 2024] libceph: osd8 up
[Mon Aug 26 17:47:17 2024] TCP: request_sock_TCP: Possible SYN flooding on port
8443. Sending cookies. Check SNMP counters.
[Tue Aug 27 13:11:49 2024] perf: interrupt took too long (5708 > 5606),
lowering kernel.perf_event_max_sample_rate to 35000
[Tue Aug 27 21:47:54 2024] ------------[ cut here ]------------
[Tue Aug 27 21:47:54 2024] Attempt to access reserved inode number 0x100
[Tue Aug 27 21:47:54 2024] WARNING: CPU: 116 PID: 3346898 at
fs/ceph/super.h:546 __lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:47:54 2024] Modules linked in: ceph libceph fscache netfs
xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user nft_counter xt_addrtype
nft_compat nf_tables nfnetlink br_netfilter bridge nvme_fabrics nvme_core
overlay intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm
8021q garp rapl mrp stp llc ipmi_ssif joydev input_leds ccp ptdma k10temp
acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler bonding tls mac_hid binfmt_misc
sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efi_pstore
ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456
async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
libcrc32c raid0 multipath linear hid_generic usbhid hid raid1 mlx4_ib ib_uverbs
ib_core ast drm_vram_helper drm_ttm_helper ttm crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel drm_kms_helper sha256_ssse3 syscopyarea ixgbe sha1_ssse3
sysfillrect sysimgblt
[Tue Aug 27 21:47:54 2024] aesni_intel fb_sys_fops crypto_simd cec cryptd igb
xfrm_algo rc_core r8169 ahci dca xhci_pci mlx4_core realtek drm mdio
i2c_algo_bit libahci i2c_piix4 xhci_pci_renesas
[Tue Aug 27 21:47:54 2024] CPU: 116 PID: 3346898 Comm: pwd Not tainted
5.15.0-117-generic #127-Ubuntu
[Tue Aug 27 21:47:54 2024] Hardware name: Supermicro Super Server/H11DSi, BIOS
2.8 12/14/2023
[Tue Aug 27 21:47:54 2024] RIP: 0010:__lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:47:54 2024] Code: 7e 2f 48 85 c0 0f 85 1d ff ff ff 48 63 c3 85
db 0f 89 51 ff ff ff e9 0d ff ff ff 4c 89 e6 48 c7 c7 a0 4c 0d c1 e8 66 68 e7
cb <0f> 0b e9 36 ff ff ff be 03 00 00 00 48 89 45 c0 e8 67 5a 7b cb 48
[Tue Aug 27 21:47:54 2024] RSP: 0018:ffffb832af8bfb18 EFLAGS: 00010286
[Tue Aug 27 21:47:54 2024] RAX: 0000000000000000 RBX: ffff98764eb64360 RCX:
0000000000000027
[Tue Aug 27 21:47:54 2024] RDX: ffff98b44e720588 RSI: 0000000000000001 RDI:
ffff98b44e720580
[Tue Aug 27 21:47:54 2024] RBP: ffffb832af8bfb58 R08: 0000000000000003 R09:
0000000000000100
[Tue Aug 27 21:47:54 2024] R10: 0000000000ffff10 R11: 000000000000000f R12:
0000000000000100
[Tue Aug 27 21:47:54 2024] R13: ffff98367972d800 R14: ffff98367972d8a8 R15:
ffff9877f0895b08
[Tue Aug 27 21:47:54 2024] FS: 00007fb709d76740(0000)
GS:ffff98b44e700000(0000) knlGS:0000000000000000
[Tue Aug 27 21:47:54 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Tue Aug 27 21:47:54 2024] CR2: 00007fb709f4a733 CR3: 000000443554a000 CR4:
0000000000350ee0
[Tue Aug 27 21:47:54 2024] Call Trace:
[Tue Aug 27 21:47:54 2024] <TASK>
[Tue Aug 27 21:47:54 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:47:54 2024] ? show_trace_log_lvl+0x28e/0x2ea
[Tue Aug 27 21:47:54 2024] ? show_trace_log_lvl+0x28e/0x2ea
[Tue Aug 27 21:47:54 2024] ? ceph_lookup_inode+0xe/0x40 [ceph]
[Tue Aug 27 21:47:54 2024] ? show_regs.part.0+0x23/0x29
[Tue Aug 27 21:47:54 2024] ? show_regs.cold+0x8/0xd
[Tue Aug 27 21:47:54 2024] ? __lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:47:54 2024] ? __warn+0x8c/0x100
[Tue Aug 27 21:47:54 2024] ? __lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:47:54 2024] ? report_bug+0xa4/0xd0
[Tue Aug 27 21:47:54 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:47:54 2024] ? handle_bug+0x39/0x90
[Tue Aug 27 21:47:54 2024] ? exc_invalid_op+0x19/0x70
[Tue Aug 27 21:47:54 2024] ? asm_exc_invalid_op+0x1b/0x20
[Tue Aug 27 21:47:54 2024] ? __lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:47:54 2024] ? __lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:47:54 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:47:54 2024] ? kmem_cache_alloc_trace+0x19e/0x2e0
[Tue Aug 27 21:47:54 2024] ceph_lookup_inode+0xe/0x40 [ceph]
[Tue Aug 27 21:47:54 2024] lookup_quotarealm_inode+0x170/0x230 [ceph]
[Tue Aug 27 21:47:54 2024] ceph_quota_is_max_bytes_exceeded+0x5d/0x70 [ceph]
[Tue Aug 27 21:47:54 2024] ceph_write_iter+0x196/0x7a0 [ceph]
[Tue Aug 27 21:47:54 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:47:54 2024] ? __alloc_pages+0x17e/0x330
[Tue Aug 27 21:47:54 2024] new_sync_write+0x114/0x1a0
[Tue Aug 27 21:47:54 2024] ? __entry_text_end+0xe419/0x10251d
[Tue Aug 27 21:47:54 2024] vfs_write+0x1d5/0x270
[Tue Aug 27 21:47:54 2024] ksys_write+0x67/0xf0
[Tue Aug 27 21:47:54 2024] __x64_sys_write+0x19/0x20
[Tue Aug 27 21:47:54 2024] x64_sys_call+0x47c/0x1fa0
[Tue Aug 27 21:47:54 2024] do_syscall_64+0x56/0xb0
[Tue Aug 27 21:47:54 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:47:54 2024] ? handle_mm_fault+0xd8/0x2c0
[Tue Aug 27 21:47:54 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:47:54 2024] ? do_user_addr_fault+0x1e7/0x670
[Tue Aug 27 21:47:54 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:47:54 2024] ? exit_to_user_mode_prepare+0x37/0xb0
[Tue Aug 27 21:47:54 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:47:54 2024] ? irqentry_exit_to_user_mode+0xe/0x20
[Tue Aug 27 21:47:54 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:47:54 2024] ? irqentry_exit+0x1d/0x30
[Tue Aug 27 21:47:54 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:47:54 2024] ? exc_page_fault+0x89/0x170
[Tue Aug 27 21:47:54 2024] entry_SYSCALL_64_after_hwframe+0x6c/0xd6
[Tue Aug 27 21:47:54 2024] RIP: 0033:0x7fb709e8d887
[Tue Aug 27 21:47:54 2024] Code: 10 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb
b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f
05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[Tue Aug 27 21:47:54 2024] RSP: 002b:00007fffd7c919e8 EFLAGS: 00000246
ORIG_RAX: 0000000000000001
[Tue Aug 27 21:47:54 2024] RAX: ffffffffffffffda RBX: 000000000000000a RCX:
00007fb709e8d887
[Tue Aug 27 21:47:54 2024] RDX: 000000000000000a RSI: 00007fffd7c91b90 RDI:
0000000000000002
[Tue Aug 27 21:47:54 2024] RBP: 00007fffd7c91b90 R08: 000000000000000a R09:
000000007fffffff
[Tue Aug 27 21:47:54 2024] R10: 0000000000000000 R11: 0000000000000246 R12:
000000000000000a
[Tue Aug 27 21:47:54 2024] R13: 00007fb709f946a0 R14: 00007fb709f8fa00 R15:
000000000000000a
[Tue Aug 27 21:47:54 2024] </TASK>
[Tue Aug 27 21:47:54 2024] ---[ end trace 4a8cb305b28f5d85 ]---
[Tue Aug 27 21:48:55 2024] ------------[ cut here ]------------
[Tue Aug 27 21:48:55 2024] Attempt to access reserved inode number 0x100
[Tue Aug 27 21:48:55 2024] WARNING: CPU: 223 PID: 3349411 at
fs/ceph/super.h:546 __lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:48:55 2024] Modules linked in: ceph libceph fscache netfs
xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user nft_counter xt_addrtype
nft_compat nf_tables nfnetlink br_netfilter bridge nvme_fabrics nvme_core
overlay intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm
8021q garp rapl mrp stp llc ipmi_ssif joydev input_leds ccp ptdma k10temp
acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler bonding tls mac_hid binfmt_misc
sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efi_pstore
ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456
async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
libcrc32c raid0 multipath linear hid_generic usbhid hid raid1 mlx4_ib ib_uverbs
ib_core ast drm_vram_helper drm_ttm_helper ttm crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel drm_kms_helper sha256_ssse3 syscopyarea ixgbe sha1_ssse3
sysfillrect sysimgblt
[Tue Aug 27 21:48:55 2024] aesni_intel fb_sys_fops crypto_simd cec cryptd igb
xfrm_algo rc_core r8169 ahci dca xhci_pci mlx4_core realtek drm mdio
i2c_algo_bit libahci i2c_piix4 xhci_pci_renesas
[Tue Aug 27 21:48:55 2024] CPU: 223 PID: 3349411 Comm: sh Tainted: G W
5.15.0-117-generic #127-Ubuntu
[Tue Aug 27 21:48:55 2024] Hardware name: Supermicro Super Server/H11DSi, BIOS
2.8 12/14/2023
[Tue Aug 27 21:48:55 2024] RIP: 0010:__lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:48:55 2024] Code: 7e 2f 48 85 c0 0f 85 1d ff ff ff 48 63 c3 85
db 0f 89 51 ff ff ff e9 0d ff ff ff 4c 89 e6 48 c7 c7 a0 4c 0d c1 e8 66 68 e7
cb <0f> 0b e9 36 ff ff ff be 03 00 00 00 48 89 45 c0 e8 67 5a 7b cb 48
[Tue Aug 27 21:48:55 2024] RSP: 0018:ffffb832b14e78a8 EFLAGS: 00010286
[Tue Aug 27 21:48:55 2024] RAX: 0000000000000000 RBX: ffff9877f0895ae0 RCX:
0000000000000027
[Tue Aug 27 21:48:55 2024] RDX: ffff98b44f1e0588 RSI: 0000000000000001 RDI:
ffff98b44f1e0580
[Tue Aug 27 21:48:55 2024] RBP: ffffb832b14e78e8 R08: 0000000000000003 R09:
0000000000000100
[Tue Aug 27 21:48:55 2024] R10: 0000000000ffff10 R11: 000000000000000f R12:
0000000000000100
[Tue Aug 27 21:48:55 2024] R13: ffff98367972d800 R14: ffff98367972d8a8 R15:
ffff9877f0895b08
[Tue Aug 27 21:48:55 2024] FS: 00007f9a948e9740(0000)
GS:ffff98b44f1c0000(0000) knlGS:0000000000000000
[Tue Aug 27 21:48:55 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Tue Aug 27 21:48:55 2024] CR2: 00005643ccf09ab8 CR3: 0000004463a36000 CR4:
0000000000350ee0
[Tue Aug 27 21:48:55 2024] Call Trace:
[Tue Aug 27 21:48:55 2024] <TASK>
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? show_trace_log_lvl+0x28e/0x2ea
[Tue Aug 27 21:48:55 2024] ? show_trace_log_lvl+0x28e/0x2ea
[Tue Aug 27 21:48:55 2024] ? ceph_lookup_inode+0xe/0x40 [ceph]
[Tue Aug 27 21:48:55 2024] ? show_regs.part.0+0x23/0x29
[Tue Aug 27 21:48:55 2024] ? show_regs.cold+0x8/0xd
[Tue Aug 27 21:48:55 2024] ? __lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:48:55 2024] ? __warn+0x8c/0x100
[Tue Aug 27 21:48:55 2024] ? __lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:48:55 2024] ? report_bug+0xa4/0xd0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? handle_bug+0x39/0x90
[Tue Aug 27 21:48:55 2024] ? exc_invalid_op+0x19/0x70
[Tue Aug 27 21:48:55 2024] ? asm_exc_invalid_op+0x1b/0x20
[Tue Aug 27 21:48:55 2024] ? __lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:48:55 2024] ? __lookup_inode+0x164/0x1a0 [ceph]
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ceph_lookup_inode+0xe/0x40 [ceph]
[Tue Aug 27 21:48:55 2024] lookup_quotarealm_inode+0x170/0x230 [ceph]
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ceph_quota_is_max_bytes_exceeded+0x5d/0x70 [ceph]
[Tue Aug 27 21:48:55 2024] ceph_write_iter+0x196/0x7a0 [ceph]
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? filename_lookup+0xcf/0x1d0
[Tue Aug 27 21:48:55 2024] new_sync_write+0x114/0x1a0
[Tue Aug 27 21:48:55 2024] ? __entry_text_end+0xe419/0x10251d
[Tue Aug 27 21:48:55 2024] vfs_write+0x1d5/0x270
[Tue Aug 27 21:48:55 2024] ksys_write+0x67/0xf0
[Tue Aug 27 21:48:55 2024] __x64_sys_write+0x19/0x20
[Tue Aug 27 21:48:55 2024] x64_sys_call+0x47c/0x1fa0
[Tue Aug 27 21:48:55 2024] do_syscall_64+0x56/0xb0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? page_counter_cancel+0x2e/0x80
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? page_counter_uncharge+0x22/0x40
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? drain_stock+0x6d/0xb0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? refill_stock+0xa2/0xb0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? __mod_memcg_lruvec_state+0x67/0xf0
[Tue Aug 27 21:48:55 2024] ? __alloc_pages+0x17e/0x330
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? __mod_lruvec_state+0x37/0x50
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? __mod_lruvec_page_state+0x6b/0xb0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? lru_cache_add_inactive_or_unevictable+0x2e/0xe0
[Tue Aug 27 21:48:55 2024] ? page_add_new_anon_rmap+0x69/0x100
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? set_pte+0x9/0x20
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? do_anonymous_page+0x1f9/0x3c0
[Tue Aug 27 21:48:55 2024] ? __do_sys_getcwd+0x13b/0x1c0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? kmem_cache_free+0x24f/0x290
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? __do_sys_getcwd+0x13b/0x1c0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? exit_to_user_mode_prepare+0x37/0xb0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? syscall_exit_to_user_mode+0x2c/0x50
[Tue Aug 27 21:48:55 2024] ? x64_sys_call+0x1f88/0x1fa0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? do_syscall_64+0x63/0xb0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? do_user_addr_fault+0x1e7/0x670
[Tue Aug 27 21:48:55 2024] ? exit_to_user_mode_prepare+0x37/0xb0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? exit_to_user_mode_prepare+0x37/0xb0
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? irqentry_exit_to_user_mode+0xe/0x20
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? irqentry_exit+0x1d/0x30
[Tue Aug 27 21:48:55 2024] ? srso_return_thunk+0x5/0x10
[Tue Aug 27 21:48:55 2024] ? exc_page_fault+0x89/0x170
[Tue Aug 27 21:48:55 2024] entry_SYSCALL_64_after_hwframe+0x6c/0xd6
[Tue Aug 27 21:48:55 2024] RIP: 0033:0x7f9a94a00887
[Tue Aug 27 21:48:55 2024] Code: 10 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb
b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f
05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[Tue Aug 27 21:48:55 2024] RSP: 002b:00007ffd7a829aa8 EFLAGS: 00000246
ORIG_RAX: 0000000000000001
[Tue Aug 27 21:48:55 2024] RAX: ffffffffffffffda RBX: 0000000000000007 RCX:
00007f9a94a00887
[Tue Aug 27 21:48:55 2024] RDX: 0000000000000007 RSI: 00005643ccf08ab8 RDI:
0000000000000002
[Tue Aug 27 21:48:55 2024] RBP: 00005643cc5c2040 R08: 0000000000000000 R09:
00007ffd7a829a10
[Tue Aug 27 21:48:55 2024] R10: 0000000000000000 R11: 0000000000000246 R12:
0000000000000007
[Tue Aug 27 21:48:55 2024] R13: 0000000000000002 R14: 00007ffd7a829b60 R15:
00005643ccf08ab8
[Tue Aug 27 21:48:55 2024] </TASK>
[Tue Aug 27 21:48:55 2024] ---[ end trace 4a8cb305b28f5d86 ]---
[Wed Aug 28 03:24:42 2024] ceph: mds0 caps stale
[Wed Aug 28 03:25:12 2024] ceph: mds0 caps renewed
[Wed Aug 28 05:33:50 2024] ceph: mds0 caps renewed
[Wed Aug 28 10:16:02 2024] ceph: mds0 caps stale
[Wed Aug 28 10:16:13 2024] ceph: mds0 caps renewed
[Wed Aug 28 12:55:41 2024] ceph: mds0 caps stale
[Wed Aug 28 12:55:53 2024] ceph: mds0 caps renewed
[Wed Aug 28 14:00:31 2024] perf: interrupt took too long (7185 > 7135),
lowering kernel.perf_event_max_sample_rate to 27750
[Wed Aug 28 17:41:39 2024] ceph: mds0 caps stale
[Wed Aug 28 17:41:46 2024] ceph: mds0 caps renewed
[Wed Aug 28 21:07:15 2024] libceph: mds1 (1)192.168.30.20:6801 socket closed
(con state OPEN)
[Wed Aug 28 21:07:19 2024] libceph: mds0 (1)192.168.30.21:6801 socket closed
(con state OPEN)
[Wed Aug 28 21:07:20 2024] ceph: mds0 caps stale
[Wed Aug 28 21:07:20 2024] libceph: mds0 (1)192.168.30.21:6801 session reset
[Wed Aug 28 21:07:20 2024] libceph: mds1 (1)192.168.30.20:6801 session reset
[Wed Aug 28 21:07:20 2024] ceph: mds1 closed our session
[Wed Aug 28 21:07:20 2024] ceph: mds1 reconnect start
[Wed Aug 28 21:07:20 2024] ceph: mds1 reconnect denied
[Wed Aug 28 21:07:20 2024] ceph: mds0 closed our session
[Wed Aug 28 21:07:20 2024] ceph: mds0 reconnect start
[Wed Aug 28 21:07:25 2024] libceph: mds1 (1)192.168.30.20:6801 socket closed
(con state OPEN)
[Wed Aug 28 21:07:28 2024] libceph: mds1 (1)192.168.30.20:6801 session reset
[Wed Aug 28 21:07:28 2024] ceph: mds1 closed our session
[Wed Aug 28 21:07:28 2024] ceph: mds1 reconnect start
[Wed Aug 28 21:07:28 2024] ceph: mds1 reconnect denied
[Wed Aug 28 21:07:28 2024] libceph: mds1 (1)192.168.30.20:6801 socket closed
(con state V1_CONNECT_MSG)
[Wed Aug 28 21:07:28 2024] ceph: mds1 rejected session
[Wed Aug 28 21:08:18 2024] ceph: mds0 reconnect denied
[Wed Aug 28 21:08:18 2024] remove_session_caps_cb: 8467 callbacks suppressed
[Wed Aug 28 21:08:18 2024] ceph: dropping dirty+flushing Fx state for
0000000008378aeb 1099605119345
[Wed Aug 28 21:08:18 2024] ceph: dropping dirty+flushing Fx state for
00000000957bf224 1099605119328
[Wed Aug 28 21:08:18 2024] ceph: dropping dirty+flushing Fx state for
00000000bf6e0482 1099605119511
[Wed Aug 28 21:08:18 2024] ceph: dropping dirty+flushing Fx state for
0000000082828846 1099605119766
[Wed Aug 28 21:08:18 2024] ceph: dropping dirty+flushing Fx state for
00000000bdb8be99 1099605119533
[Wed Aug 28 21:08:18 2024] ceph: dropping dirty+flushing Fx state for
00000000ea4ae75b 1099605119913
[Wed Aug 28 21:08:18 2024] ceph: dropping dirty+flushing Fx state for
000000007e55ae2f 1099605119377
[Wed Aug 28 21:08:18 2024] ceph: dropping dirty+flushing Fx state for
000000001db3b3fc 1099605119400
[Wed Aug 28 21:08:18 2024] ceph: dropping dirty+flushing Fx state for
00000000ef27c2ec 1099605119387
[Wed Aug 28 21:08:18 2024] ceph: dropping dirty+flushing Fx state for
00000000579088a5 1099605119432
[Wed Aug 28 21:08:22 2024] libceph: mds0 (1)192.168.30.21:6801 socket closed
(con state V1_CONNECT_MSG)
[Wed Aug 28 21:08:22 2024] ceph: mds0 rejected session
[Wed Aug 28 21:37:28 2024] libceph: mds1 (1)192.168.30.20:6801 socket closed
(con state OPEN)
[Wed Aug 28 21:38:22 2024] libceph: mds0 (1)192.168.30.21:6801 socket closed
(con state OPEN)
root@regio:~#
Could you help?
Thanks
** Affects: ceph (Ubuntu)
Importance: Undecided
Status: New
** Attachment added: "dmesg.20240828_2237.tar.gz"
https://bugs.launchpad.net/bugs/2078272/+attachment/5810187/+files/dmesg.20240828_2237.tar.gz
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2078272
Title:
CephFS frequently becomes unavailable mounted to /home
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2078272/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
