[Kernel-packages] [Bug 1205639] [NEW] bcmwl-kernel-source 6.20.155.1+bdcom-0ubuntu6: bcmwl kernel module failed to build
Public bug reported: DKMS make.log for bcmwl-6.20.155.1+bdcom for kernel 3.10.2-031002-generic (x86_64) samedi 27 juillet 2013, 15:16:56 (UTC+0200) make: entrant dans le répertoire « /usr/src/linux-headers-3.10.2-031002-generic » CFG80211 API is prefered for this kernel version Using CFG80211 API LD /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/built-in.o CC [M] /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/shared/linux_osl.o CC [M] /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.o /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c: In function ‘wl_tkip_printstats’: /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2704:7: attention : passing argument 1 of ‘wl->tkipmodops->print_stats’ from incompatible pointer type [enabled by default] /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2704:7: note: expected ‘struct seq_file *’ but argument is of type ‘char *’ /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2707:4: attention : passing argument 1 of ‘wl->tkipmodops->print_stats’ from incompatible pointer type [enabled by default] /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2707:4: note: expected ‘struct seq_file *’ but argument is of type ‘char *’ /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c: In function ‘wl_reg_proc_entry’: /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2917:2: erreur: implicit declaration of function ‘create_proc_entry’ [-Werror=implicit-function-declaration] /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2917:22: attention : assignment makes pointer from integer without a cast [enabled by default] /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2922:16: erreur: déréférencement d'un pointeur de type incomplet /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2923:16: erreur: déréférencement d'un pointeur de type incomplet /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2924:16: erreur: déréférencement d'un pointeur de type incomplet cc1: some warnings being treated as errors make[1]: *** [/var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.o] Erreur 1 make: *** [_module_/var/lib/dkms/bcmwl/6.20.155.1+bdcom/build] Erreur 2 make: quittant le répertoire « /usr/src/linux-headers-3.10.2-031002-generic » ProblemType: Package DistroRelease: Ubuntu 13.04 Package: bcmwl-kernel-source 6.20.155.1+bdcom-0ubuntu6 Uname: Linux 3.8.8-030808-generic x86_64 NonfreeKernelModules: wl ApportVersion: 2.9.2-0ubuntu8.3 Architecture: amd64 DKMSKernelVersion: 3.10.2-031002-generic Date: Sat Jul 27 15:17:00 2013 InstallationDate: Installed on 2013-02-28 (148 days ago) InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5) MarkForUpload: True PackageVersion: 6.20.155.1+bdcom-0ubuntu6 SourcePackage: bcmwl Title: bcmwl-kernel-source 6.20.155.1+bdcom-0ubuntu6: bcmwl kernel module failed to build UpgradeStatus: Upgraded to raring on 2013-04-29 (89 days ago) ** Affects: bcmwl (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-package need-duplicate-check raring -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to bcmwl in Ubuntu. https://bugs.launchpad.net/bugs/1205639 Title: bcmwl-kernel-source 6.20.155.1+bdcom-0ubuntu6: bcmwl kernel module failed to build Status in “bcmwl” package in Ubuntu: New Bug description: DKMS make.log for bcmwl-6.20.155.1+bdcom for kernel 3.10.2-031002-generic (x86_64) samedi 27 juillet 2013, 15:16:56 (UTC+0200) make: entrant dans le répertoire « /usr/src/linux-headers-3.10.2-031002-generic » CFG80211 API is prefered for this kernel version Using CFG80211 API LD /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/built-in.o CC [M] /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/shared/linux_osl.o CC [M] /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.o /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c: In function ‘wl_tkip_printstats’: /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2704:7: attention : passing argument 1 of ‘wl->tkipmodops->print_stats’ from incompatible pointer type [enabled by default] /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2704:7: note: expected ‘struct seq_file *’ but argument is of type ‘char *’ /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2707:4: attention : passing argument 1 of ‘wl->tkipmodops->print_stats’ from incompatible pointer type [enabled by default] /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2707:4: note: expected ‘struct seq_file *’ but argument is of type ‘char *’ /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c: In function ‘wl_reg_proc_entry’: /var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/src/wl/sys/wl_linux.c:2917:2: erreur: impl
[Kernel-packages] [Bug 1842447] Re: Kernel Panic with linux-image-4.15.0-60-generic when specifying nameserver in docker-compose
I agree with Taher (in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842447/comments/15), this bug seems to impact a lot of systems (my colo host was kernel panic restarting about every 75-90 minutes, all weekend). It has a NAT firewall on it (for the hosted VMs), but no Docker/Wireguard, etc. My guess for the 75-90 minutes is that is how long it took to dirty enough memory that the relevant value just happened not to already be 0 (NULL). (Curiously I installed -60 on Thursday last week, and the first issue didn't happen until Friday, so it's been worse over the weekend than the first 24 hours. I enabled kernel.panic=15 after the first issue, to automate recovery, and was *extremely* glad I did so.) Honestly I'd suggest withdrawing -60 as it's very unstable in a lot of common configurations. And also suggest expediting the release of -62, which AFAICT just contains the one line fix for the bug in -60. Now it's Monday morning (and thus I can get into the colo if needed), I've upgraded the colo system to the proposed -62 version, and crossing my fingers the system is more stable as a result. In case it helps others, I found I needed to: (a) https://wiki.ubuntu.com/Testing/EnableProposed (changing "xenial" to "bionic", for 18.04 LTS, including enabling the low priority pin of bionic-proposed); and (b) sudo apt-get install linux-generic/bionic-proposed linux-signed- generic/bionic-proposed linux-headers-generic/bionic-proposed (without at least two of those three, the proposed update metapackages wouldn't install due to conflicts; I'm not sure if linux-signed-generic is needed, but it's still installed, so I chose to keep it in sync.) That list of packages found by looking for 4.15.0-60 versioned packages that didn't have that version in their package name (ie, to find the generic metapackages). Ewen PS: Reboots (due to kernel panic, and kernel.panic=15 sysctl) over the weekend: -=- cut here -=- ewen@naosr620:~$ last | grep reboot reboot system boot 4.15.0-62-generi Mon Sep 9 10:43 still running reboot system boot 4.15.0-60-generi Mon Sep 9 10:14 - 10:39 (00:25) reboot system boot 4.15.0-60-generi Mon Sep 9 08:48 - 10:09 (01:21) reboot system boot 4.15.0-60-generi Mon Sep 9 07:33 - 10:09 (02:36) reboot system boot 4.15.0-60-generi Mon Sep 9 06:18 - 10:09 (03:51) reboot system boot 4.15.0-60-generi Mon Sep 9 05:03 - 10:09 (05:06) reboot system boot 4.15.0-60-generi Mon Sep 9 03:48 - 10:09 (06:21) reboot system boot 4.15.0-60-generi Mon Sep 9 02:33 - 10:09 (07:36) reboot system boot 4.15.0-60-generi Mon Sep 9 01:13 - 10:09 (08:56) reboot system boot 4.15.0-60-generi Sun Sep 8 23:58 - 10:09 (10:11) reboot system boot 4.15.0-60-generi Sun Sep 8 22:43 - 10:09 (11:26) reboot system boot 4.15.0-60-generi Sun Sep 8 21:28 - 10:09 (12:41) reboot system boot 4.15.0-60-generi Sun Sep 8 20:08 - 10:09 (14:01) reboot system boot 4.15.0-60-generi Sun Sep 8 18:53 - 10:09 (15:16) reboot system boot 4.15.0-60-generi Sun Sep 8 17:38 - 10:09 (16:31) reboot system boot 4.15.0-60-generi Sun Sep 8 16:23 - 10:09 (17:46) reboot system boot 4.15.0-60-generi Sun Sep 8 15:08 - 10:09 (19:01) reboot system boot 4.15.0-60-generi Sun Sep 8 13:53 - 10:09 (20:16) reboot system boot 4.15.0-60-generi Sun Sep 8 12:29 - 10:09 (21:40) reboot system boot 4.15.0-60-generi Sun Sep 8 11:14 - 10:09 (22:55) reboot system boot 4.15.0-60-generi Sun Sep 8 09:57 - 10:09 (1+00:12) reboot system boot 4.15.0-60-generi Sun Sep 8 08:43 - 10:09 (1+01:26) reboot system boot 4.15.0-60-generi Sun Sep 8 07:28 - 10:09 (1+02:41) reboot system boot 4.15.0-60-generi Sun Sep 8 06:13 - 10:09 (1+03:56) reboot system boot 4.15.0-60-generi Sun Sep 8 04:54 - 10:09 (1+05:15) reboot system boot 4.15.0-60-generi Sun Sep 8 03:34 - 10:09 (1+06:35) reboot system boot 4.15.0-60-generi Sun Sep 8 02:18 - 10:09 (1+07:51) reboot system boot 4.15.0-60-generi Sun Sep 8 01:03 - 10:09 (1+09:06) reboot system boot 4.15.0-60-generi Sat Sep 7 23:48 - 10:09 (1+10:21) reboot system boot 4.15.0-60-generi Sat Sep 7 22:55 - 10:09 (1+11:14) reboot system boot 4.15.0-60-generi Sat Sep 7 22:34 - 10:09 (1+11:35) reboot system boot 4.15.0-60-generi Sat Sep 7 21:19 - 10:09 (1+12:50) reboot system boot 4.15.0-60-generi Sat Sep 7 20:03 - 10:09 (1+14:06) reboot system boot 4.15.0-60-generi Sat Sep 7 18:48 - 10:09 (1+15:21) reboot system boot 4.15.0-60-generi Sat Sep 7 17:33 - 10:09 (1+16:36) reboot system boot 4.15.0-60-generi Sat Sep 7 16:18 - 10:09 (1+17:51) reboot system boot 4.15.0-60-generi Sat Sep 7 15:03 - 10:09 (1+19:06) reboot system boot 4.15.0-60-generi Sat Sep 7 13:42 - 10:09 (1+20:27) reboot system boot 4.15.0-60-generi Sat Sep 7 12:27 - 10:09 (1+21:42) reboot system boot 4.15.0-60-generi Sat Sep 7 11:12 - 10:09 (1+22:57) re
[Kernel-packages] [Bug 1842447] Re: Kernel Panic with linux-image-4.15.0-60-generic when specifying nameserver in docker-compose
FTR, I think this is the fix in -62: https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/commit/?h=master- next&id=b502cfeffec81be8564189e5498fd3f252b27900 and it appears to be the only change from -60 to -62: -=- cut here -=- ewen@naosr620:~$ zcat /usr/share/doc/linux-headers-4.15.0-62-generic/changelog.Debian.gz | head -9 linux (4.15.0-62.69) bionic; urgency=medium * bionic/linux: 4.15.0-62.69 -proposed tracker (LP: #1842746) * Kernel Panic with linux-image-4.15.0-60-generic when specifying nameserver in docker-compose (LP: #1842447) - ip: frags: fix crash in ip_do_fragment() -- Khalid Elmously Wed, 04 Sep 2019 16:11:43 -0400 ewen@naosr620:~$ -=- cut here -=- It's a one line fix. Ewen -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842447 Title: Kernel Panic with linux-image-4.15.0-60-generic when specifying nameserver in docker-compose Status in linux package in Ubuntu: Confirmed Status in linux source package in Bionic: Fix Committed Bug description: [Impact] Some fragmentation+NAT workloads will cause kernel BUG/Ooops. [Test case] sudo iptables -t nat -I POSTROUTING -j MASQUERADE sudo hping3 192.168.122.1 -s 1000 -p 2000 -d 6 [Regression potential] This could make fragmented packets stop flowing. So, make sure fragmented pings still work. ping 192.168.122.1 -s 6 still works, even with the above nat rule. Hello, there are multiple inquries in the mailcow GitHub issues over at https://github.com/mailcow/mailcow-dockerized/issues/2904 that the latest kernel linux-image-4.15.0-60-generic causes kernel panics when "- dns" setting is used within the docker-compose.yml file, for yet some unclear reasons. Multiple users on different systems (e.g. virtualized ones on VMware ESXi and KVM) were able to reproduce the same issue. I was also able to reproduce this constantly on a completely new deployed Ubuntu 18.04 VM (KVM) with a fresh mailcow installation. Steps to reproduce: 1. Install a clean Ubuntu 18.04(.03) machine 2. Upgrade the installation to linux-image-4.15.0-60-generic 3. Setup mailcow as instructed at https://mailcow.github.io/mailcow-dockerized-docs/i_u_m_install/ (just takes less than a minute, easy to reproduce) 4. Start mailcow with "dns"-settings specified in docker-compose file (Make sure using the older docker-compose version with dns settings: https://raw.githubusercontent.com/mailcow/mailcow-dockerized/a1403b7a5969637df23001d05c59c2a20774fbb5/docker-compose.yml) 5. Wait a few minutes, then kernel crash appears Using this workaround it appears to be stable again: https://github.com/mailcow/mailcow- dockerized/commit/dc6eea5142c063e26408a685b66fbb7754408ec2 I've attached the apport file to this bug. Please let me know if you need any kind of further information. (As this is my first bug report here, I hope I have included all required information helping you finding the cause.) Kind regards, Patrik To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842447/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1843152] Re: Kernel Panic with linux kernel 4.15.0-60 possibly related to network subsystem
These symptoms sound very much like https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842447 (I found the bug I'm commenting on while searching for additional links about the issue in 1842447). There's a -62 kernel in proposed updates which hopefully contains the fix for this bug. See https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842447 for hints on how to install the proposed update kernel. So far the trigger is sounding like NAT + 4.15.0-60 kernel + sufficient time that the relevant uninitialised variable is not clean from boot. I think this is the fix in -62: https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/commit/?h=master- next&id=b502cfeffec81be8564189e5498fd3f252b27900 Ewen -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1843152 Title: Kernel Panic with linux kernel 4.15.0-60 possibly related to network subsystem Status in linux package in Ubuntu: Confirmed Bug description: 1. Releases: Ubuntu Server 16.04 and Ubuntu Server 18.04 2. Package: Linux Kernel 4.15.0-60 amd64 3. What I expected to happen: Not a recurrent kernel panic. 4. What happened instead: Recurrent kernel panic. While running Ubuntu 16.04 server on an internet appliance that was serving as my router, after the system was updated from kernel 4.15.0-58 to 4.15.0-60, I began getting frequent kernel panics (system seldom remained up for more than an hour after a reboot). I have included example stack traces below. After memtest revealed no problems, and fsck revealed no problems, I opted to rebuild from a fresh install, this time of Ubuntu 18.04, server. Upon completing the rebuilt, and put it back in service. It crashed with the same kernel panic within 30 minutes. I have since updated to kernel 5.0.0-27 and the kernel panics have completely stopped. [10170.296117] kernel BUG at /build/linux-5mCauq/linux-4.15.0/net/ipv4/ip_output.c:636! [10170.304214] invalid opcode: [#1] SMP PTI [10170.308692] Modules linked in: st lp parport_pc ppdev parport ipt_REJECT nf_reject_ipv4 xt_condition(OE) xt_time xt_comment xt_iface(OE) xt_multiport xt_conntrack xt_set xt_recent xt_hashlimit xt_addrtype xt_mark iptable_mangle xt_nat xt_REDIRECT nf_nat_redirect ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_iprange iptable_nat nf_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp xt_CT iptable_raw ip_set_list_set ip_set_hash_ip ip_set_hash_net ip_set_hash_mac ip_set xt_NFLOG nfnetlink_log nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_sane nf_conntrack_netlink nfnetlink nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netbios_ns [10170.381810] nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp ts_kmp nf_conntrack_amanda nf_conntrack iptable_filter bridge intel_rapl intel_soc_dts_thermal intel_soc_dts_iosf intel_powerclamp coretemp kvm_intel snd_hda_codec_hdmi kvm irqbypass punit_atom_debug intel_cstate snd_hda_intel snd_hda_codec snd_hda_core joydev snd_hwdep snd_pcm snd_timer serio_raw input_leds snd lpc_ich soundcore shpchp mac_hid mei_txe mei sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 8021q garp mrp stp llc ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit drm_kms_helper [10170.454089] ghash_clmulni_intel e1000e syscopyarea sysfillrect sysimgblt fb_sys_fops ptp ahci psmouse cryptd drm pps_core libahci video hid_generic usbhid hid [last unloaded: parport_pc] [10170.471175] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G OE 4.15.0-60-generic #67-Ubuntu [10170.480269] Hardware name: Protectli FW1/FW1, BIOS 5.6.5 05/14/2019 [10170.486704] RIP: 0010:ip_do_fragment+0x482/0x820 [10170.491462] RSP: 0018:92533fc83a18 EFLAGS: 00010202 [10170.496824] RAX: 0001 RBX: 92532b43ed00 RCX: 8d64cdf0 [10170.504158] RDX: 0024 RSI: 05c8 RDI: 925329f06300 [10170.511485] RBP: 92533fc83a80 R08: 925330be9700 R09: 05dc [10170.518777] R10: R11: 92533fc839d0 R12: 0014 [10170.526096] R13: 92532dea4300 R14: 2328 R15: 925330be974e [10170.533430] FS: () GS:92533fc8() knlGS: [10170.541729] CS: 0010 DS: ES: CR0: 80050033 [10170.547631] CR2: 7f3a9dd13b80 CR3: 9640a000 CR4: 001006e0 [10170.554957] Call Trace: [10170.557452] [10170.559541] ? ip_copy_metadata+0x220/0x220
[Kernel-packages] [Bug 1842447] Re: Kernel Panic with linux-image-4.15.0-60-generic when specifying nameserver in docker-compose
FTR, 4.15.0-62 seems *much* better than 4.15.0-60. With 4.15.0-60 this system was kernel panic restarting every 75-90 minutes; now it's been up since I installed 4.15.0-62, over 5 hours ago: -=- cut here -=- ewen@naosr620:~$ uname -r 4.15.0-62-generic ewen@naosr620:~$ uptime 16:09:54 up 5:26, 1 user, load average: 0.24, 0.25, 0.24 ewen@naosr620:~$ -=- cut here -=- That one line fix seems important :-) Ewen -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842447 Title: Kernel Panic with linux-image-4.15.0-60-generic when specifying nameserver in docker-compose Status in linux package in Ubuntu: Confirmed Status in linux source package in Bionic: Fix Committed Bug description: [Impact] Some fragmentation+NAT workloads will cause kernel BUG/Ooops. [Test case] sudo iptables -t nat -I POSTROUTING -j MASQUERADE sudo hping3 192.168.122.1 -s 1000 -p 2000 -d 6 [Regression potential] This could make fragmented packets stop flowing. So, make sure fragmented pings still work. ping 192.168.122.1 -s 6 still works, even with the above nat rule. Hello, there are multiple inquries in the mailcow GitHub issues over at https://github.com/mailcow/mailcow-dockerized/issues/2904 that the latest kernel linux-image-4.15.0-60-generic causes kernel panics when "- dns" setting is used within the docker-compose.yml file, for yet some unclear reasons. Multiple users on different systems (e.g. virtualized ones on VMware ESXi and KVM) were able to reproduce the same issue. I was also able to reproduce this constantly on a completely new deployed Ubuntu 18.04 VM (KVM) with a fresh mailcow installation. Steps to reproduce: 1. Install a clean Ubuntu 18.04(.03) machine 2. Upgrade the installation to linux-image-4.15.0-60-generic 3. Setup mailcow as instructed at https://mailcow.github.io/mailcow-dockerized-docs/i_u_m_install/ (just takes less than a minute, easy to reproduce) 4. Start mailcow with "dns"-settings specified in docker-compose file (Make sure using the older docker-compose version with dns settings: https://raw.githubusercontent.com/mailcow/mailcow-dockerized/a1403b7a5969637df23001d05c59c2a20774fbb5/docker-compose.yml) 5. Wait a few minutes, then kernel crash appears Using this workaround it appears to be stable again: https://github.com/mailcow/mailcow- dockerized/commit/dc6eea5142c063e26408a685b66fbb7754408ec2 I've attached the apport file to this bug. Please let me know if you need any kind of further information. (As this is my first bug report here, I hope I have included all required information helping you finding the cause.) Kind regards, Patrik To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842447/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2018960] Re: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0
** Summary changed: - linux-image-5.4.0-149-generic (focal-proposed): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 + linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed in Ubuntu. https://bugs.launchpad.net/bugs/2018960 Title: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 Status in linux-signed package in Ubuntu: Confirmed Bug description: After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu Focal), I noticed that it was suddenly getting a bunch of kernel log (dmesg) reports like: WARNING: CPU: 4 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 while investigating I determined that it is currently running the focal-proposed kernel (linux-image-5.4.0-149-generic), which it turns out was enabled for this server (clearly it seemed like a good idea at the time). I'm not expecting focal-proposed to be fixed as if it were a release package, but since I couldn't find any reports on Launchpad I figured I should let y'all know this focal-proposed package could do with some additional work before it's actually released :-) There have been at least 80 such reports in the last 5 hours since the server was rebooted, differing only by the CPU core and the process reported, although it seems the last one was a couple of hours ago, so I guess it's traffic dependent/timing dependent. ewen@naosr620:~$ uptime 16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06 ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID: [0-9]*/PID: N/;' | uniq -c 88 WARNING: CPU: N PID: N at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 ewen@naosr620:~$ Ubuntu Release: ewen@naosr620:~$ lsb_release -rd Description: Ubuntu 20.04.6 LTS Release: 20.04 ewen@naosr620:~$ Kernel/package version affected: ewen@naosr620:~$ uname -a Linux naosr620 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ewen@naosr620:~$ dpkg -l | grep linux-image | grep 149 ii linux-image-5.4.0-149-generic 5.4.0-149.166 amd64Signed kernel image generic ii linux-image-generic5.4.0.149.147 amd64 Generic Linux kernel image ewen@naosr620:~$ apt-cache policy linux-image-5.4.0-149-generic linux-image-5.4.0-149-generic: Installed: 5.4.0-149.166 Candidate: 5.4.0-149.166 Version table: *** 5.4.0-149.166 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status ewen@naosr620:~$ apt-cache policy linux-image-generic linux-image-generic: Installed: 5.4.0.149.147 Candidate: 5.4.0.149.147 Version table: *** 5.4.0.149.147 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status 5.4.0.148.146 500 500 https://mirror.fsmg.org.nz/ubuntu focal-updates/main amd64 Packages 500 https://mirror.fsmg.org.nz/ubuntu focal-security/main amd64 Packages 5.4.0.26.32 500 500 https://mirror.fsmg.org.nz/ubuntu focal/main amd64 Packages ewen@naosr620:~$ ewen@naosr620:~$ apt-cache show linux-image-5.4.0-149-generic | grep Source: Source: linux-signed ewen@naosr620:~$ Full example dmesg, including stack trace (they all seem to be WARNINGs, and other than filling dmesg / system logs the system "appears to be running okay", so I'm not going to rush another reboot now -- near end of business day): ewen@naosr620:~$ date Tue 09 May 2023 16:34:56 NZST ewen@naosr620:~$ dmesg -T | tail -100 | grep -B 150 "end trace" | grep -A 999 "cut here" [Tue May 9 14:21:18 2023] [ cut here ] [Tue May 9 14:21:18 2023] WARNING: CPU: 10 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 [Tue May 9 14:21:18 2023] Modules linked in: mpt3sas raid_class scsi_transport_sas mptctl mptbase vhost_net vhost tap ip6t_REJECT nf_reject_ipv6 ip6table_mangle ip6table_nat ip6table_raw nf_log_ipv6 xt_recent ipt_REJECT nf_reject_ipv4 xt_hashlimit xt_addrtype xt_multiport xt_comment xt_conntrack xt_mark iptable_mangle xt_MASQUERADE iptable_nat xt_CT xt_tcpudp iptable_raw nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_irc ebtable_filter nf_nat_h323 ebtables nf_nat_ftp nf_nat_amanda ts_kmp ip6table_filter nf_conntrack_amanda nf_nat ip6_tables nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink nfnetlink nf_conntrack
[Kernel-packages] [Bug 2018960] Re: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0
As Steve Matos notes, linux-image-5.4.0-149-generic seems to have been released into the main focal-updates repository, apparently without making an attempt to address the regression that causes the "net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0". So I have retitled the bug with "regression" instead of "focal- proposed". As an additional report, I did downgrade the server I was having problems with to linux-image-5.4.0-148-generic, about two weeks ago. I've had *zero* dmesg warnings since then. Ie, 148 has zero reports in 13 days; 149 was getting at least 5-20 WARNING "net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0" reports per day. So I conclude the "net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0" was introduced bewteen 5.4.0-148.165 and 5.4.0-149.166, and appears to be a regression in behaviour, in a long term stable release update. I'm *not* planning to reboot onto 5.4.0.149.147 again on that server (production, remote colo), and will at least wait for the next kernel release. But ideally someone at Canonical/Ubuntu would actually investigate the cause of this regression in behaviour. (I don't have time at present, due to my day job being excessively busy, to try to troubleshoot what Ubuntu changed between 5.4.0-148.165 and 5.4.0-149.166; but if the problem remains unfixed in a month when I'll hopefully have some more time I might try to review the diff between those two versions to hopefully identify a likely cause.) Ewen -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed in Ubuntu. https://bugs.launchpad.net/bugs/2018960 Title: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 Status in linux-signed package in Ubuntu: Confirmed Bug description: After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu Focal), I noticed that it was suddenly getting a bunch of kernel log (dmesg) reports like: WARNING: CPU: 4 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 while investigating I determined that it is currently running the focal-proposed kernel (linux-image-5.4.0-149-generic), which it turns out was enabled for this server (clearly it seemed like a good idea at the time). I'm not expecting focal-proposed to be fixed as if it were a release package, but since I couldn't find any reports on Launchpad I figured I should let y'all know this focal-proposed package could do with some additional work before it's actually released :-) There have been at least 80 such reports in the last 5 hours since the server was rebooted, differing only by the CPU core and the process reported, although it seems the last one was a couple of hours ago, so I guess it's traffic dependent/timing dependent. ewen@naosr620:~$ uptime 16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06 ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID: [0-9]*/PID: N/;' | uniq -c 88 WARNING: CPU: N PID: N at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 ewen@naosr620:~$ Ubuntu Release: ewen@naosr620:~$ lsb_release -rd Description: Ubuntu 20.04.6 LTS Release: 20.04 ewen@naosr620:~$ Kernel/package version affected: ewen@naosr620:~$ uname -a Linux naosr620 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ewen@naosr620:~$ dpkg -l | grep linux-image | grep 149 ii linux-image-5.4.0-149-generic 5.4.0-149.166 amd64Signed kernel image generic ii linux-image-generic5.4.0.149.147 amd64Generic Linux kernel image ewen@naosr620:~$ apt-cache policy linux-image-5.4.0-149-generic linux-image-5.4.0-149-generic: Installed: 5.4.0-149.166 Candidate: 5.4.0-149.166 Version table: *** 5.4.0-149.166 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status ewen@naosr620:~$ apt-cache policy linux-image-generic linux-image-generic: Installed: 5.4.0.149.147 Candidate: 5.4.0.149.147 Version table: *** 5.4.0.149.147 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status 5.4.0.148.146 500 500 https://mirror.fsmg.org.nz/ubuntu focal-updates/main amd64 Packages 500 https://mirror.fsmg.org.nz/ubuntu focal-security/main amd64 Packages 5.4.0.26.32 500 500 https://mirror.fsmg.org.nz/ubuntu focal/main amd64 Packages ewen@naosr620:~$ ewen@naosr620:~$ apt-cache show linux-image-5.4.0-149-generic | grep Source: Source: linux-signed ewen@naosr620:~$ Full example dmesg, including stack trace (they al
[Kernel-packages] [Bug 2018960] Re: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0
Based on the comments above, and the Oracle Linux report that Leif M found (thanks!) it seems likely that this is an upstream Linux kernel "stable" backport which (probably in the upstream Linux kernel "stable") was incomplete. Which now a bunch of Linux distros (including Ubuntu and Oracle Linux) have imported. (I really wish the upstream Linux kernel "stable" process had a more thorough process than just randomly cherry picking patches made to later kernels and hoping for the best :-/ ) >From the Oracle Linux report (https://github.com/oracle/linux- uek/issues/15 -- found by Leif M above) it looks like one of the triggering factors is that the Linux kernel is doing NAT. That's also consistent with my experience -- the server with problems hosts several KVM virtual machines, and has a NAT firewall in front of them. But I've not seen similar reports on a laptop or a desktop running the 149 kernel, which don't normally run any virtual machines or do NAT themselves. Some of the patches linked from the Oracle Linux bug report also seem to suggest that in certain control flows one of the values that's being relied on to be a pointer actually isn't used/used as a pointer (presumably hence 0). Which makes me think "improperly back ported" (most likely by the Linux kernel "stable" process) is the most likely cause, and a state check got missed. Ewen -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed in Ubuntu. https://bugs.launchpad.net/bugs/2018960 Title: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 Status in linux-signed package in Ubuntu: Confirmed Status in linux-signed-kvm package in Ubuntu: New Bug description: After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu Focal), I noticed that it was suddenly getting a bunch of kernel log (dmesg) reports like: WARNING: CPU: 4 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 while investigating I determined that it is currently running the focal-proposed kernel (linux-image-5.4.0-149-generic), which it turns out was enabled for this server (clearly it seemed like a good idea at the time). I'm not expecting focal-proposed to be fixed as if it were a release package, but since I couldn't find any reports on Launchpad I figured I should let y'all know this focal-proposed package could do with some additional work before it's actually released :-) There have been at least 80 such reports in the last 5 hours since the server was rebooted, differing only by the CPU core and the process reported, although it seems the last one was a couple of hours ago, so I guess it's traffic dependent/timing dependent. ewen@naosr620:~$ uptime 16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06 ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID: [0-9]*/PID: N/;' | uniq -c 88 WARNING: CPU: N PID: N at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 ewen@naosr620:~$ Ubuntu Release: ewen@naosr620:~$ lsb_release -rd Description: Ubuntu 20.04.6 LTS Release: 20.04 ewen@naosr620:~$ Kernel/package version affected: ewen@naosr620:~$ uname -a Linux naosr620 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ewen@naosr620:~$ dpkg -l | grep linux-image | grep 149 ii linux-image-5.4.0-149-generic 5.4.0-149.166 amd64Signed kernel image generic ii linux-image-generic5.4.0.149.147 amd64Generic Linux kernel image ewen@naosr620:~$ apt-cache policy linux-image-5.4.0-149-generic linux-image-5.4.0-149-generic: Installed: 5.4.0-149.166 Candidate: 5.4.0-149.166 Version table: *** 5.4.0-149.166 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status ewen@naosr620:~$ apt-cache policy linux-image-generic linux-image-generic: Installed: 5.4.0.149.147 Candidate: 5.4.0.149.147 Version table: *** 5.4.0.149.147 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status 5.4.0.148.146 500 500 https://mirror.fsmg.org.nz/ubuntu focal-updates/main amd64 Packages 500 https://mirror.fsmg.org.nz/ubuntu focal-security/main amd64 Packages 5.4.0.26.32 500 500 https://mirror.fsmg.org.nz/ubuntu focal/main amd64 Packages ewen@naosr620:~$ ewen@naosr620:~$ apt-cache show linux-image-5.4.0-149-generic | grep Source: Source: linux-signed ewen@naosr620:~$ Full example dmesg, including stack trace (they all seem to be WARNINGs, and other than filling dmes
[Kernel-packages] [Bug 2018960] Re: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0
Thanks to Steve Matos for pointing out that the relevant warning was
likely reverted in the Linux Kernel "Stable" release 5.4.234, which is
tracked in Ubuntu as:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2017691
>From digging into the Linux kernel stable git history (below) I'm fairly
sure that the issue should have been fixed in Linux 5.4.0-151 (and thus
fixed in 5.4.0.152), by entirely removing the warning line that was
triggering. It also seems like the (known) triggering cause was IPv6
traffic, which is at least consistent with my problematic server (and
explains why it wasn't seen often enough for Ubuntu / Canonical do
engage with this bug).
More details below.
The Linux Kernel mailing list announcement of Linux stable kernel 5.4.234 is:
https://lore.kernel.org/lkml/1677842623227...@kroah.com/T/
and a patch stream list:
https://lore.kernel.org/all/20230301180651.177668...@linuxfoundation.org/
which includes a diff from (Linux kernel upstream) 5.4.233 to 5.4.234.
And that diff removes the line:
WARN_ON(sk->sk_forward_alloc);
from net/core/stream.c. (It's *not* a warn on once, so I've no idea why
the revert says that; it was a warn on every occurence, which explains
why we got a bunch of them.)
>From upstream Linux stable 5.4.234 that relevant warning does seem to be
explicitly removed:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-
rc.git/log/net/core/stream.c?h=v5.4.234
by the commit:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-
rc.git/commit/net/core/stream.c?h=v5.4.234&id=ee8cd3abe7228161be143702efa3d03a65a757c8
which gives the background:
"Christoph Paasch reported that commit b5fc29233d28 ("inet6: Remove
inet6_destroy_sock() in sk->sk_prot->destroy().") started triggering
WARN_ON_ONCE(sk->sk_forward_alloc) in sk_stream_kill_queues(). [0 - 2]
Also, we can reproduce it by a program in [3]."
So apparently a change in IPv6 handling started triggering this
particular problem in the cleanup phase elsewhere. And they've now
removed the warning we were getting entirely (which it seems has been
there a while longer, but not triggerd until now, and wasn't needed at
all as there were other checks on that data elsewhere).
This seems to be the upstream Linux kernel bug report:
https://lore.kernel.org/netdev/39725ab4-88f1-41b3-b07f-949c5caef...@icloud.com/
and the relevant bug referenced from there:
https://github.com/multipath-tcp/mptcp_net-next/issues/341
both from February 2023, with the fix released March 2023
(https://gitlab.com/linux-
kernel/stable/-/commit/a103859aaa718cf13cb5f55c3a33512dbab613f7). Yet
Ubuntu Linux released the buggy version in April 2023 :-/
Anyway I'm fairly confident that Ubuntu 5.4.0-152 should be okay. I'll
update my server later this month, and if there are no more reports by
about mid July 2023 I'll close this issue.
Ewen
** Bug watch added: github.com/multipath-tcp/mptcp_net-next/issues #341
https://github.com/multipath-tcp/mptcp_net-next/issues/341
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-signed in Ubuntu.
https://bugs.launchpad.net/bugs/2018960
Title:
linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
Status in linux-signed package in Ubuntu:
Confirmed
Status in linux-signed-kvm package in Ubuntu:
Confirmed
Bug description:
After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu
Focal), I noticed that it was suddenly getting a bunch of kernel log
(dmesg) reports like:
WARNING: CPU: 4 PID: 0 at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
while investigating I determined that it is currently running the
focal-proposed kernel (linux-image-5.4.0-149-generic), which it turns
out was enabled for this server (clearly it seemed like a good idea at
the time).
I'm not expecting focal-proposed to be fixed as if it were a release
package, but since I couldn't find any reports on Launchpad I figured
I should let y'all know this focal-proposed package could do with some
additional work before it's actually released :-)
There have been at least 80 such reports in the last 5 hours since the
server was rebooted, differing only by the CPU core and the process
reported, although it seems the last one was a couple of hours ago, so
I guess it's traffic dependent/timing dependent.
ewen@naosr620:~$ uptime
16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06
ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID:
[0-9]*/PID: N/;' | uniq -c
88 WARNING: CPU: N PID: N at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
ewen@naosr620:~$
Ubuntu Release:
ewen@naosr620:~$ lsb_release -rd
Description: Ubuntu 20.04.6 LTS
Release:
[Kernel-packages] [Bug 2018960] [NEW] linux-image-5.4.0-149-generic (focal-proposed): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0
Public bug reported:
After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu
Focal), I noticed that it was suddenly getting a bunch of kernel log
(dmesg) reports like:
WARNING: CPU: 4 PID: 0 at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
while investigating I determined that it is currently running the focal-
proposed kernel (linux-image-5.4.0-149-generic), which it turns out was
enabled for this server (clearly it seemed like a good idea at the
time).
I'm not expecting focal-proposed to be fixed as if it were a release
package, but since I couldn't find any reports on Launchpad I figured I
should let y'all know this focal-proposed package could do with some
additional work before it's actually released :-)
There have been at least 80 such reports in the last 5 hours since the
server was rebooted, differing only by the CPU core and the process
reported, although it seems the last one was a couple of hours ago, so I
guess it's traffic dependent/timing dependent.
ewen@naosr620:~$ uptime
16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06
ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID:
[0-9]*/PID: N/;' | uniq -c
88 WARNING: CPU: N PID: N at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
ewen@naosr620:~$
Ubuntu Release:
ewen@naosr620:~$ lsb_release -rd
Description:Ubuntu 20.04.6 LTS
Release:20.04
ewen@naosr620:~$
Kernel/package version affected:
ewen@naosr620:~$ uname -a
Linux naosr620 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023
x86_64 x86_64 x86_64 GNU/Linux
ewen@naosr620:~$ dpkg -l | grep linux-image | grep 149
ii linux-image-5.4.0-149-generic 5.4.0-149.166
amd64Signed kernel image generic
ii linux-image-generic5.4.0.149.147
amd64Generic Linux kernel image
ewen@naosr620:~$ apt-cache policy linux-image-5.4.0-149-generic
linux-image-5.4.0-149-generic:
Installed: 5.4.0-149.166
Candidate: 5.4.0-149.166
Version table:
*** 5.4.0-149.166 500
500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages
100 /var/lib/dpkg/status
ewen@naosr620:~$ apt-cache policy linux-image-generic
linux-image-generic:
Installed: 5.4.0.149.147
Candidate: 5.4.0.149.147
Version table:
*** 5.4.0.149.147 500
500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages
100 /var/lib/dpkg/status
5.4.0.148.146 500
500 https://mirror.fsmg.org.nz/ubuntu focal-updates/main amd64 Packages
500 https://mirror.fsmg.org.nz/ubuntu focal-security/main amd64 Packages
5.4.0.26.32 500
500 https://mirror.fsmg.org.nz/ubuntu focal/main amd64 Packages
ewen@naosr620:~$
ewen@naosr620:~$ apt-cache show linux-image-5.4.0-149-generic | grep Source:
Source: linux-signed
ewen@naosr620:~$
Full example dmesg, including stack trace (they all seem to be WARNINGs, and
other than filling dmesg / system logs the system "appears to be running okay",
so I'm not going to rush another reboot now -- near end of business day):
ewen@naosr620:~$ date
Tue 09 May 2023 16:34:56 NZST
ewen@naosr620:~$ dmesg -T | tail -100 | grep -B 150 "end trace" | grep -A 999
"cut here"
[Tue May 9 14:21:18 2023] [ cut here ]
[Tue May 9 14:21:18 2023] WARNING: CPU: 10 PID: 0 at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
[Tue May 9 14:21:18 2023] Modules linked in: mpt3sas raid_class
scsi_transport_sas mptctl mptbase vhost_net vhost tap ip6t_REJECT
nf_reject_ipv6 ip6table_mangle ip6table_nat ip6table_raw nf_log_ipv6 xt_recent
ipt_REJECT nf_reject_ipv4 xt_hashlimit xt_addrtype xt_multiport xt_comment
xt_conntrack xt_mark iptable_mangle xt_MASQUERADE iptable_nat xt_CT xt_tcpudp
iptable_raw nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp
nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_irc
ebtable_filter nf_nat_h323 ebtables nf_nat_ftp nf_nat_amanda ts_kmp
ip6table_filter nf_conntrack_amanda nf_nat ip6_tables nf_conntrack_sane
nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink
nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc
nf_conntrack_h323 nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
iptable_filter bpfilter dell_rbu nls_iso8859_1 ipmi_ssif input_leds joydev
cdc_ether usbnet mii cdc_acm intel_rapl_msr
intel_rapl_common
[Tue May 9 14:21:18 2023] sb_edac x86_pkg_temp_thermal intel_powerclamp
binfmt_misc coretemp dcdbas kvm_intel kvm rapl intel_cstate mei_me mei ipmi_si
ipmi_devintf mac_hid ipmi_msghandler acpi_power_meter 8021q garp mrp bridge stp
llc sch_fq_codel ramoops reed_solomon efi_pstore ip_tables x_tables autofs4
btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq
async_xor async_tx xor raid6_pq libcrc32c raid1 raid
[Kernel-packages] [Bug 2018960] Re: linux-image-5.4.0-149-generic (focal-proposed): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0
FTR, I figured out how focal-proposed ended up enabled, with packages intalled from focal-proposed: the system had bionic-proposed enabled, but pinned to be low priority (so packages could be installed from proposed by manual override), at some point in the past. After upgrade to focal, the apt sources proposed line was updated to focal-proposed, but the pin (preferences.d) was not updated to focal-proposed being low priority (it still listed bionic-proposed as low priority, which was irrelevant after the upgrade to focal). This led to a number of packages being installed from focal-proposed, including the kernel images. I've explicitly downgraded a few of the less system critical ones (X11, update manager, etc), but will probably have to let the other 30-40 packages eventually get replaced by newer versions from newer Ubuntu Focal stable releases. Also FTR, the reports above (WARNING: CPU: 10 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0) have continued overnight, but coming in bursts -- some hours had no reports, and other hours had 5-10 reports. So definitely feels like a race condition of some sort. Ewen -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed in Ubuntu. https://bugs.launchpad.net/bugs/2018960 Title: linux-image-5.4.0-149-generic (focal-proposed): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 Status in linux-signed package in Ubuntu: New Bug description: After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu Focal), I noticed that it was suddenly getting a bunch of kernel log (dmesg) reports like: WARNING: CPU: 4 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 while investigating I determined that it is currently running the focal-proposed kernel (linux-image-5.4.0-149-generic), which it turns out was enabled for this server (clearly it seemed like a good idea at the time). I'm not expecting focal-proposed to be fixed as if it were a release package, but since I couldn't find any reports on Launchpad I figured I should let y'all know this focal-proposed package could do with some additional work before it's actually released :-) There have been at least 80 such reports in the last 5 hours since the server was rebooted, differing only by the CPU core and the process reported, although it seems the last one was a couple of hours ago, so I guess it's traffic dependent/timing dependent. ewen@naosr620:~$ uptime 16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06 ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID: [0-9]*/PID: N/;' | uniq -c 88 WARNING: CPU: N PID: N at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 ewen@naosr620:~$ Ubuntu Release: ewen@naosr620:~$ lsb_release -rd Description: Ubuntu 20.04.6 LTS Release: 20.04 ewen@naosr620:~$ Kernel/package version affected: ewen@naosr620:~$ uname -a Linux naosr620 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ewen@naosr620:~$ dpkg -l | grep linux-image | grep 149 ii linux-image-5.4.0-149-generic 5.4.0-149.166 amd64Signed kernel image generic ii linux-image-generic5.4.0.149.147 amd64 Generic Linux kernel image ewen@naosr620:~$ apt-cache policy linux-image-5.4.0-149-generic linux-image-5.4.0-149-generic: Installed: 5.4.0-149.166 Candidate: 5.4.0-149.166 Version table: *** 5.4.0-149.166 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status ewen@naosr620:~$ apt-cache policy linux-image-generic linux-image-generic: Installed: 5.4.0.149.147 Candidate: 5.4.0.149.147 Version table: *** 5.4.0.149.147 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status 5.4.0.148.146 500 500 https://mirror.fsmg.org.nz/ubuntu focal-updates/main amd64 Packages 500 https://mirror.fsmg.org.nz/ubuntu focal-security/main amd64 Packages 5.4.0.26.32 500 500 https://mirror.fsmg.org.nz/ubuntu focal/main amd64 Packages ewen@naosr620:~$ ewen@naosr620:~$ apt-cache show linux-image-5.4.0-149-generic | grep Source: Source: linux-signed ewen@naosr620:~$ Full example dmesg, including stack trace (they all seem to be WARNINGs, and other than filling dmesg / system logs the system "appears to be running okay", so I'm not going to rush another reboot now -- near end of business day): ewen@naosr620:~$ date Tue 09 May 2023 16:34:56 NZST ewen@naosr620:~$ dmesg -T | tail -100 | grep -B 150 "end trace" | grep -A 999 "cut here" [Tue May 9 14:21:18 2023] ---
[Kernel-packages] [Bug 2018960] Re: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0
I have seen no further reports since upgrading to linux-image-5.4.0-152. So given the analysis above, I believe this bug was introduced by the (upstream) Linux Kernel Stable Team, due to a change in freeing memory that didn't take this assertion into account. And then was later fixed by the (upstream) Linux Kernel Stable Team. It appears to only affect IPv6 packets, in certain situations (forwarding them?). >From the reports above, it appears to be fixed in linux-image-5.4.0-151 / linux-image-5.4.0-152, and in linux-image-kvm-1093. Which is consistent with the (upstream) Linux Kernel Stable Team fixes being copied out to various kernel versions (and then pulled into Ubuntu's kernel updates). So I'm going to close the issue as "fixed released" for both linux-image and linux-image-kvm. I believe the fix was released about 1 month ago (mid June 2023). Ewen -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed in Ubuntu. https://bugs.launchpad.net/bugs/2018960 Title: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 Status in linux-signed package in Ubuntu: Fix Released Status in linux-signed-kvm package in Ubuntu: Fix Released Bug description: After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu Focal), I noticed that it was suddenly getting a bunch of kernel log (dmesg) reports like: WARNING: CPU: 4 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 while investigating I determined that it is currently running the focal-proposed kernel (linux-image-5.4.0-149-generic), which it turns out was enabled for this server (clearly it seemed like a good idea at the time). I'm not expecting focal-proposed to be fixed as if it were a release package, but since I couldn't find any reports on Launchpad I figured I should let y'all know this focal-proposed package could do with some additional work before it's actually released :-) There have been at least 80 such reports in the last 5 hours since the server was rebooted, differing only by the CPU core and the process reported, although it seems the last one was a couple of hours ago, so I guess it's traffic dependent/timing dependent. ewen@naosr620:~$ uptime 16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06 ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID: [0-9]*/PID: N/;' | uniq -c 88 WARNING: CPU: N PID: N at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 ewen@naosr620:~$ Ubuntu Release: ewen@naosr620:~$ lsb_release -rd Description: Ubuntu 20.04.6 LTS Release: 20.04 ewen@naosr620:~$ Kernel/package version affected: ewen@naosr620:~$ uname -a Linux naosr620 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ewen@naosr620:~$ dpkg -l | grep linux-image | grep 149 ii linux-image-5.4.0-149-generic 5.4.0-149.166 amd64Signed kernel image generic ii linux-image-generic5.4.0.149.147 amd64Generic Linux kernel image ewen@naosr620:~$ apt-cache policy linux-image-5.4.0-149-generic linux-image-5.4.0-149-generic: Installed: 5.4.0-149.166 Candidate: 5.4.0-149.166 Version table: *** 5.4.0-149.166 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status ewen@naosr620:~$ apt-cache policy linux-image-generic linux-image-generic: Installed: 5.4.0.149.147 Candidate: 5.4.0.149.147 Version table: *** 5.4.0.149.147 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status 5.4.0.148.146 500 500 https://mirror.fsmg.org.nz/ubuntu focal-updates/main amd64 Packages 500 https://mirror.fsmg.org.nz/ubuntu focal-security/main amd64 Packages 5.4.0.26.32 500 500 https://mirror.fsmg.org.nz/ubuntu focal/main amd64 Packages ewen@naosr620:~$ ewen@naosr620:~$ apt-cache show linux-image-5.4.0-149-generic | grep Source: Source: linux-signed ewen@naosr620:~$ Full example dmesg, including stack trace (they all seem to be WARNINGs, and other than filling dmesg / system logs the system "appears to be running okay", so I'm not going to rush another reboot now -- near end of business day): ewen@naosr620:~$ date Tue 09 May 2023 16:34:56 NZST ewen@naosr620:~$ dmesg -T | tail -100 | grep -B 150 "end trace" | grep -A 999 "cut here" [Tue May 9 14:21:18 2023] [ cut here ] [Tue May 9 14:21:18 2023] WARNING: CPU: 10 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 [Tue May 9 14:21:18 2023] Module
[Kernel-packages] [Bug 2018960] Re: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0
Fixed in linux-image-5.4.0-151 and later, by pulling in upstream Linux
Kernel Stable Team fix.
** Changed in: linux-signed (Ubuntu)
Status: Confirmed => Fix Released
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-signed in Ubuntu.
https://bugs.launchpad.net/bugs/2018960
Title:
linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
Status in linux-signed package in Ubuntu:
Fix Released
Status in linux-signed-kvm package in Ubuntu:
Fix Released
Bug description:
After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu
Focal), I noticed that it was suddenly getting a bunch of kernel log
(dmesg) reports like:
WARNING: CPU: 4 PID: 0 at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
while investigating I determined that it is currently running the
focal-proposed kernel (linux-image-5.4.0-149-generic), which it turns
out was enabled for this server (clearly it seemed like a good idea at
the time).
I'm not expecting focal-proposed to be fixed as if it were a release
package, but since I couldn't find any reports on Launchpad I figured
I should let y'all know this focal-proposed package could do with some
additional work before it's actually released :-)
There have been at least 80 such reports in the last 5 hours since the
server was rebooted, differing only by the CPU core and the process
reported, although it seems the last one was a couple of hours ago, so
I guess it's traffic dependent/timing dependent.
ewen@naosr620:~$ uptime
16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06
ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID:
[0-9]*/PID: N/;' | uniq -c
88 WARNING: CPU: N PID: N at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
ewen@naosr620:~$
Ubuntu Release:
ewen@naosr620:~$ lsb_release -rd
Description: Ubuntu 20.04.6 LTS
Release: 20.04
ewen@naosr620:~$
Kernel/package version affected:
ewen@naosr620:~$ uname -a
Linux naosr620 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023
x86_64 x86_64 x86_64 GNU/Linux
ewen@naosr620:~$ dpkg -l | grep linux-image | grep 149
ii linux-image-5.4.0-149-generic 5.4.0-149.166
amd64Signed kernel image generic
ii linux-image-generic5.4.0.149.147
amd64 Generic Linux kernel image
ewen@naosr620:~$ apt-cache policy linux-image-5.4.0-149-generic
linux-image-5.4.0-149-generic:
Installed: 5.4.0-149.166
Candidate: 5.4.0-149.166
Version table:
*** 5.4.0-149.166 500
500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64
Packages
100 /var/lib/dpkg/status
ewen@naosr620:~$ apt-cache policy linux-image-generic
linux-image-generic:
Installed: 5.4.0.149.147
Candidate: 5.4.0.149.147
Version table:
*** 5.4.0.149.147 500
500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64
Packages
100 /var/lib/dpkg/status
5.4.0.148.146 500
500 https://mirror.fsmg.org.nz/ubuntu focal-updates/main amd64
Packages
500 https://mirror.fsmg.org.nz/ubuntu focal-security/main amd64
Packages
5.4.0.26.32 500
500 https://mirror.fsmg.org.nz/ubuntu focal/main amd64 Packages
ewen@naosr620:~$
ewen@naosr620:~$ apt-cache show linux-image-5.4.0-149-generic | grep Source:
Source: linux-signed
ewen@naosr620:~$
Full example dmesg, including stack trace (they all seem to be WARNINGs, and
other than filling dmesg / system logs the system "appears to be running okay",
so I'm not going to rush another reboot now -- near end of business day):
ewen@naosr620:~$ date
Tue 09 May 2023 16:34:56 NZST
ewen@naosr620:~$ dmesg -T | tail -100 | grep -B 150 "end trace" | grep -A 999
"cut here"
[Tue May 9 14:21:18 2023] [ cut here ]
[Tue May 9 14:21:18 2023] WARNING: CPU: 10 PID: 0 at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
[Tue May 9 14:21:18 2023] Modules linked in: mpt3sas raid_class
scsi_transport_sas mptctl mptbase vhost_net vhost tap ip6t_REJECT
nf_reject_ipv6 ip6table_mangle ip6table_nat ip6table_raw nf_log_ipv6 xt_recent
ipt_REJECT nf_reject_ipv4 xt_hashlimit xt_addrtype xt_multiport xt_comment
xt_conntrack xt_mark iptable_mangle xt_MASQUERADE iptable_nat xt_CT xt_tcpudp
iptable_raw nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp
nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_irc
ebtable_filter nf_nat_h323 ebtables nf_nat_ftp nf_nat_amanda ts_kmp
ip6table_filter nf_conntrack_amanda nf_nat ip6_tables nf_conntrack_sane
nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink
nfnetlink nf_conntr
[Kernel-packages] [Bug 2018960] Re: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0
Fixed in linux-image-kvm-1093 by pulling in upstream Linux Kernel Stable
Team fix.
** Changed in: linux-signed-kvm (Ubuntu)
Status: Confirmed => Fix Released
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-signed in Ubuntu.
https://bugs.launchpad.net/bugs/2018960
Title:
linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
Status in linux-signed package in Ubuntu:
Fix Released
Status in linux-signed-kvm package in Ubuntu:
Fix Released
Bug description:
After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu
Focal), I noticed that it was suddenly getting a bunch of kernel log
(dmesg) reports like:
WARNING: CPU: 4 PID: 0 at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
while investigating I determined that it is currently running the
focal-proposed kernel (linux-image-5.4.0-149-generic), which it turns
out was enabled for this server (clearly it seemed like a good idea at
the time).
I'm not expecting focal-proposed to be fixed as if it were a release
package, but since I couldn't find any reports on Launchpad I figured
I should let y'all know this focal-proposed package could do with some
additional work before it's actually released :-)
There have been at least 80 such reports in the last 5 hours since the
server was rebooted, differing only by the CPU core and the process
reported, although it seems the last one was a couple of hours ago, so
I guess it's traffic dependent/timing dependent.
ewen@naosr620:~$ uptime
16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06
ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID:
[0-9]*/PID: N/;' | uniq -c
88 WARNING: CPU: N PID: N at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
ewen@naosr620:~$
Ubuntu Release:
ewen@naosr620:~$ lsb_release -rd
Description: Ubuntu 20.04.6 LTS
Release: 20.04
ewen@naosr620:~$
Kernel/package version affected:
ewen@naosr620:~$ uname -a
Linux naosr620 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023
x86_64 x86_64 x86_64 GNU/Linux
ewen@naosr620:~$ dpkg -l | grep linux-image | grep 149
ii linux-image-5.4.0-149-generic 5.4.0-149.166
amd64Signed kernel image generic
ii linux-image-generic5.4.0.149.147
amd64 Generic Linux kernel image
ewen@naosr620:~$ apt-cache policy linux-image-5.4.0-149-generic
linux-image-5.4.0-149-generic:
Installed: 5.4.0-149.166
Candidate: 5.4.0-149.166
Version table:
*** 5.4.0-149.166 500
500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64
Packages
100 /var/lib/dpkg/status
ewen@naosr620:~$ apt-cache policy linux-image-generic
linux-image-generic:
Installed: 5.4.0.149.147
Candidate: 5.4.0.149.147
Version table:
*** 5.4.0.149.147 500
500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64
Packages
100 /var/lib/dpkg/status
5.4.0.148.146 500
500 https://mirror.fsmg.org.nz/ubuntu focal-updates/main amd64
Packages
500 https://mirror.fsmg.org.nz/ubuntu focal-security/main amd64
Packages
5.4.0.26.32 500
500 https://mirror.fsmg.org.nz/ubuntu focal/main amd64 Packages
ewen@naosr620:~$
ewen@naosr620:~$ apt-cache show linux-image-5.4.0-149-generic | grep Source:
Source: linux-signed
ewen@naosr620:~$
Full example dmesg, including stack trace (they all seem to be WARNINGs, and
other than filling dmesg / system logs the system "appears to be running okay",
so I'm not going to rush another reboot now -- near end of business day):
ewen@naosr620:~$ date
Tue 09 May 2023 16:34:56 NZST
ewen@naosr620:~$ dmesg -T | tail -100 | grep -B 150 "end trace" | grep -A 999
"cut here"
[Tue May 9 14:21:18 2023] [ cut here ]
[Tue May 9 14:21:18 2023] WARNING: CPU: 10 PID: 0 at net/core/stream.c:212
sk_stream_kill_queues+0xcf/0xe0
[Tue May 9 14:21:18 2023] Modules linked in: mpt3sas raid_class
scsi_transport_sas mptctl mptbase vhost_net vhost tap ip6t_REJECT
nf_reject_ipv6 ip6table_mangle ip6table_nat ip6table_raw nf_log_ipv6 xt_recent
ipt_REJECT nf_reject_ipv4 xt_hashlimit xt_addrtype xt_multiport xt_comment
xt_conntrack xt_mark iptable_mangle xt_MASQUERADE iptable_nat xt_CT xt_tcpudp
iptable_raw nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp
nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_irc
ebtable_filter nf_nat_h323 ebtables nf_nat_ftp nf_nat_amanda ts_kmp
ip6table_filter nf_conntrack_amanda nf_nat ip6_tables nf_conntrack_sane
nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink
nfnetlink nf_conntrack_netbios_ns nf
