Thanks for raising this. First of all I've noticed that the kernel in use is pretty outdated (package 'linux-meta-hwe-5.4 (5.4.0.90.101~18.04.80', changelog date 22 Oct 2021) and about half a year old - the current one is '5.4.0-107-generic' (package '5.4.0.107.121~18.04.92'). The delta between 5.4.0.107.121 and 5.4.0.90.101 are about 2000 commits and more than 20 are NFS related and also some about vfs. Hence I need to ask to update the system to the latest 'linux-image-generic-hwe-18.04' in 'bionic-updates' (5.4.0-107-generic) to be on the latest (and supported) level. (A test with '5.4.0.108.122~18.04.93' from 'bionic-proposed' would be ideal on top.)
It also looks like a kernel dump was created, could you please share this dump (ideally from the current kernel) for further analysis (either via IBM Box or Canonical's anon. ftp 'http://archive.admin.canonical.com/')? ** Changed in: ubuntu-z-systems Status: New => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1968096 Title: [UBUNTU 20.04] Null Pointer issue in nfs code running Ubuntu on IBM Z Status in Ubuntu on IBM z Systems: Incomplete Status in linux package in Ubuntu: New Bug description: State the component where the Bug is occuring: kernel Indicate the nature of the problem by answering the below questions: - Is this problem reproducible? No No, steps unknown, but we have seen these before - Is the system sitting at a debugger (kdb, or xmon)? No - Is the system hung? No No, dumped and rebooted - Are there any custom patches installed? Yes On base system level (CloudAppliance) we are still running with the zfpc_proc module loaded. But no recent changes in the module and is running absolutely stable in HA (same kernel and userspace, Ubuntu 20.04 LTS) - Is there any special hardware that may be relevant to this problem? Yes We are running with mlx (cloud network adapters) installed. - Is access information for the machine the problem was found on available? Yes - Is the bug occuring in a userspace application? No - Was a stack trace produced? Yes This is what mention in first comment by @Boris Barth - Did the system produce an Oops message on the console? Yes [556585.270902] illegal operation: 0001 ilc:1 [#10] SMP [556585.270905] Modules linked in: vhost_net macvtap macvlan tap rpcsec_gss_krb5 auth_rpcgss nfsv3 nfs_acl nfs lockd grace fscache veth xt_statistic ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_mark sunrpc nf_log_ipv6 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_set ip_set_hash_net ip_set_hash_ip ip_set tcp_diag inet_diag xt_comment xt_nat cls_cgroup sch_htb act_gact sch_multiq act_mirred act_pedit act_tunnel_key cls_flower act_police cls_u32 vxlan ip6_udp_tunnel udp_tunnel dummy nf_tables ebtable_filter ebtables xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key sch_ingress mlx5_ib ib_uverbs ib_core mlx5_core tls mlxfw ptp pps_core dm_integrity async_xor async_tx dm_bufio bonding xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat br_netfilter bridge vhost_vsock vmw_vsock_virtio_transport_common vhost vsock 8021q garp mrp stp llc xt_multiport xt_tcpudp qeth_l2 lcs ctcm fsm dasd_fba_mod aufs overlay scsi_dh_rdac [556585.270923] scsi_dh_emc s390_trng xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter bpfilter sch_fq_codel zFPC_proc(OE) zFPC_diag(OE) vfio_ap vfio_mdev drm vfio_iommu_type1 drm_panel_orientation_quirks i2c_core ip_tables x_tables scsi_dh_alua pkey zcrypt ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common chsc_sch qeth ccwgroup eadm_sch vfio_ccw mdev vfio btrfs libcrc32c crc32_vx_s390 xor zstd_compress raid6_pq dm_crypt virtio_blk dm_service_time dm_multipath zfcp scsi_transport_fc qdio dasd_eckd_mod dasd_mod zlib_deflate [last unloaded: tls] [556585.270945] CPU: 28 PID: 217741 Comm: worker Kdump: loaded Tainted: G D OE 5.4.0-90-generic #101-Ubuntu [556585.270947] Hardware name: IBM 8562 GT2 A00 (LPAR) [556585.270948] Krnl PSW : 0704d00180000000 0000000000000002 (0x2) [556585.270951] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [556585.270953] Krnl GPRS: 0000000000000000 0000000000000000 000003e010ebbcf8 00000071c45e1ec0 [556585.270954] 0000000000000000 0000002816f7b18c 00000078dd36a4a0 000000713a62f718 [556585.270955] 0000000000000000 000003e010ebbcf8 0000000000000068 00000071c45e1ec0 [556585.270957] 0000006090a12200 0000000000000c40 000003ff80d6fb54 000003e010ebbbf0 [556585.270959] Krnl Code:#0000000000000000: 0000 illegal >0000000000000002: 0000 illegal 0000000000000004: 0000 illegal 0000000000000006: 0000 illegal 0000000000000008: 0000 illegal 000000000000000a: 0000 illegal 000000000000000c: 0000 illegal 000000000000000e: 0000 illegal [556585.270967] Call Trace: [556585.270982] ([<000003ff80d6fb1a>] rpcauth_lookup_credcache+0x5a/0x300 [sunrpc]) [556585.270993] [<000003ff80e1182c>] nfs_ctx_key_to_expire+0xec/0x130 [nfs] [556585.271004] [<000003ff80e1189c>] nfs_key_timeout_notify+0x2c/0x70 [nfs] [556585.271014] [<000003ff80dfdf7e>] nfs_file_write+0x3e/0x320 [nfs] [556585.271016] [<00000028165944a8>] new_sync_write+0x118/0x1b0 [556585.271017] [<0000002816594ee0>] vfs_write+0xb0/0x1b0 [556585.271019] [<0000002816596a1e>] ksys_pwrite64+0x7e/0xc0 [556585.271021] [<0000002816bb26b2>] system_call+0x2a6/0x2c8 - Was a system dump produced ie kdump, netdumpmp, or LKCD? Yes That is the kdump where the stacktrace from. Enter data below to accurately describe the problem: - Problem description: Null Pointer issue in nfs code running Ubuntu Ubuntu 18.04 with HWE kernel 5.4 on IBM Z - Enter uname -a output: @lon1-qz1-sr4-rk101-s04> uname -a Linux lon1-qz1-sr4-rk101-s04 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 19:59:45 UTC 2021 s390x s390x s390x GNU/Linux - Enter failing machine type and model (ie p520 9111-520 lpar, x336 47U-8637): Manufacturer: IBM Type: 8562 Model: A00 GT2 Model Capacity: A00 00000000 Capacity Adj. Ind.: 100 LPAR CPUs Total: 16 LPAR CPUs Configured: 16 LPAR CPUs Standby: 0 LPAR CPUs Reserved: 0 LPAR CPUs Dedicated: 0 LPAR CPUs Shared: 16 LPAR CPUs G-MTID: 0 LPAR CPUs S-MTID: 1 LPAR CPUs PS-MTID: 1 - Enter primary and backup contact information (name/email): Prabhat Ranjan pranj...@in.ibm.com Christoph Schlameu? schlame...@de.ibm.com - Detail the configuration of the additonal hardware - Enter common userspace tool name: N/A - Enter name of userspace RPM: N/A - If failing tool is obtained from project website vs RPM install, what is the version/release/mod. If from the project's CVS, what is the branch tag and date of checkout (put "na" if not applicable)? N/A - Is the failing userspace tool 32-bit, 64-bit, or both? N/A - Describe how unresponsive the system is. What steps have you taken to reclaim the system: kernel oops was detected and automatically dumped and restarted - Is a debugger configured (xmon or kdb enabled)? No - Enter Oops message from console: [556585.270902] illegal operation: 0001 ilc:1 [#10] SMP [556585.270905] Modules linked in: vhost_net macvtap macvlan tap rpcsec_gss_krb5 auth_rpcgss nfsv3 nfs_acl nfs lockd grace fscache veth xt_statistic ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_mark sunrpc nf_log_ipv6 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_set ip_set_hash_net ip_set_hash_ip ip_set tcp_diag inet_diag xt_comment xt_nat cls_cgroup sch_htb act_gact sch_multiq act_mirred act_pedit act_tunnel_key cls_flower act_police cls_u32 vxlan ip6_udp_tunnel udp_tunnel dummy nf_tables ebtable_filter ebtables xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key sch_ingress mlx5_ib ib_uverbs ib_core mlx5_core tls mlxfw ptp pps_core dm_integrity async_xor async_tx dm_bufio bonding xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat br_netfilter bridge vhost_vsock vmw_vsock_virtio_transport_common vhost vsock 8021q garp mrp stp llc xt_multiport xt_tcpudp qeth_l2 lcs ctcm fsm dasd_fba_mod aufs overlay scsi_dh_rdac [556585.270923] scsi_dh_emc s390_trng xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter bpfilter sch_fq_codel zFPC_proc(OE) zFPC_diag(OE) vfio_ap vfio_mdev drm vfio_iommu_type1 drm_panel_orientation_quirks i2c_core ip_tables x_tables scsi_dh_alua pkey zcrypt ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common chsc_sch qeth ccwgroup eadm_sch vfio_ccw mdev vfio btrfs libcrc32c crc32_vx_s390 xor zstd_compress raid6_pq dm_crypt virtio_blk dm_service_time dm_multipath zfcp scsi_transport_fc qdio dasd_eckd_mod dasd_mod zlib_deflate [last unloaded: tls] [556585.270945] CPU: 28 PID: 217741 Comm: worker Kdump: loaded Tainted: G D OE 5.4.0-90-generic #101-Ubuntu [556585.270947] Hardware name: IBM 8562 GT2 A00 (LPAR) [556585.270948] Krnl PSW : 0704d00180000000 0000000000000002 (0x2) [556585.270951] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3 [556585.270953] Krnl GPRS: 0000000000000000 0000000000000000 000003e010ebbcf8 00000071c45e1ec0 [556585.270954] 0000000000000000 0000002816f7b18c 00000078dd36a4a0 000000713a62f718 [556585.270955] 0000000000000000 000003e010ebbcf8 0000000000000068 00000071c45e1ec0 [556585.270957] 0000006090a12200 0000000000000c40 000003ff80d6fb54 000003e010ebbbf0 [556585.270959] Krnl Code:#0000000000000000: 0000 illegal >0000000000000002: 0000 illegal 0000000000000004: 0000 illegal 0000000000000006: 0000 illegal 0000000000000008: 0000 illegal 000000000000000a: 0000 illegal 000000000000000c: 0000 illegal 000000000000000e: 0000 illegal [556585.270967] Call Trace: [556585.270982] ([<000003ff80d6fb1a>] rpcauth_lookup_credcache+0x5a/0x300 [sunrpc]) [556585.270993] [<000003ff80e1182c>] nfs_ctx_key_to_expire+0xec/0x130 [nfs] [556585.271004] [<000003ff80e1189c>] nfs_key_timeout_notify+0x2c/0x70 [nfs] [556585.271014] [<000003ff80dfdf7e>] nfs_file_write+0x3e/0x320 [nfs] [556585.271016] [<00000028165944a8>] new_sync_write+0x118/0x1b0 [556585.271017] [<0000002816594ee0>] vfs_write+0xb0/0x1b0 [556585.271019] [<0000002816596a1e>] ksys_pwrite64+0x7e/0xc0 [556585.271021] [<0000002816bb26b2>] system_call+0x2a6/0x2c8 - Detail the steps to reproduce this problem: unknown - Was the system configured to capture a system dump? Yes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1968096/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
