This bug is missing log files that will aid in diagnosing the problem.
While running an Ubuntu kernel (not a mainline or third-party kernel)
please enter the following command in a terminal window:
apport-collect 1886668
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable
to run this command, please add a comment stating that fact and change
the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the
Ubuntu Kernel Team.
** Changed in: linux (Ubuntu)
Status: New => Incomplete
** Tags added: bionic
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1886668
Title:
linux 4.15.0-109-generic network DoS regression vs -108
Status in linux package in Ubuntu:
Incomplete
Bug description:
Reported from a user:
Several of our infrastructure VMs recently started crashing (oops
attached), after they upgraded to -109. -108 appears to be stable.
Analysing the crash, it appears to be a wild pointer access in a BPF
filter, which makes this (probably) a network-traffic triggered crash.
[ 696.396831] general protection fault: 0000 [#1] SMP PTI
[ 696.396843] Modules linked in: iscsi_target_mod target_core_mod
ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user
xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype
iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge nfsv3 cmac
arc4 md4 rpcsec_gss_krb5 nfsv4 nls_utf8 cifs nfs aufs ccm fscache binfmt_misc
overlay xfs libcrc32c intel_rapl crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel ppdev pcbc aesni_intel aes_x86_64 crypto_simd glue_helper
cryptd input_leds joydev intel_rapl_perf serio_raw parport_pc parport mac_hid
sch_fq_codel nfsd 8021q auth_rpcgss garp nfs_acl mrp lockd stp llc grace xenfs
sunrpc xen_privcmd ip_tables x_tables autofs4 hid_generic usbhid hid psmouse
i2c_piix4 pata_acpi floppy
[ 696.396966] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 4.15.0-109-generic
#110-Ubuntu
[ 696.396979] Hardware name: Xen HVM domU, BIOS 4.7.6-1.26 12/03/2018
[ 696.396993] RIP: 0010:__cgroup_bpf_run_filter_skb+0xbb/0x1e0
[ 696.397005] RSP: 0018:ffff893fdcb83a70 EFLAGS: 00010292
[ 696.397015] RAX: 6d69546e6f697469 RBX: 0000000000000000 RCX:
0000000000000014
[ 696.397028] RDX: 0000000000000000 RSI: ffff893fd0360000 RDI:
ffff893fb5154800
[ 696.397041] RBP: ffff893fdcb83ad0 R08: 0000000000000001 R09:
0000000000000000
[ 696.397058] R10: 0000000000000000 R11: 0000000000000003 R12:
0000000000000014
[ 696.397075] R13: ffff893fb5154800 R14: 0000000000000020 R15:
ffff893fc6ba4d00
[ 696.397091] FS: 0000000000000000(0000) GS:ffff893fdcb80000(0000)
knlGS:0000000000000000
[ 696.397107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 696.397119] CR2: 000000c0001b4000 CR3: 00000006dce0a004 CR4:
00000000003606e0
[ 696.397135] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 696.397152] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 696.397169] Call Trace:
[ 696.397175] <IRQ>
[ 696.397183] sk_filter_trim_cap+0xd0/0x1b0
[ 696.397191] tcp_v4_rcv+0x8b7/0xa80
[ 696.397199] ip_local_deliver_finish+0x66/0x210
[ 696.397208] ip_local_deliver+0x7e/0xe0
[ 696.397215] ? ip_rcv_finish+0x430/0x430
[ 696.397223] ip_rcv_finish+0x129/0x430
[ 696.397230] ip_rcv+0x296/0x360
[ 696.397238] ? inet_del_offload+0x40/0x40
[ 696.397249] __netif_receive_skb_core+0x432/0xb80
[ 696.397261] ? skb_send_sock+0x50/0x50
[ 696.397271] ? tcp4_gro_receive+0x137/0x1a0
[ 696.397280] __netif_receive_skb+0x18/0x60
[ 696.397290] ? __netif_receive_skb+0x18/0x60
[ 696.397300] netif_receive_skb_internal+0x45/0xe0
[ 696.397309] napi_gro_receive+0xc5/0xf0
[ 696.397317] xennet_poll+0x9ca/0xbc0
[ 696.397325] net_rx_action+0x140/0x3a0
[ 696.397334] __do_softirq+0xe4/0x2d4
[ 696.397344] irq_exit+0xc5/0xd0
[ 696.397352] xen_evtchn_do_upcall+0x30/0x50
[ 696.397361] xen_hvm_callback_vector+0x90/0xa0
[ 696.397371] </IRQ>
[ 696.397378] RIP: 0010:native_safe_halt+0x12/0x20
[ 696.397390] RSP: 0018:ffff94c4862cbe80 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffff0c
[ 696.397405] RAX: ffffffff8efc1800 RBX: 0000000000000006 RCX:
0000000000000000
[ 696.397419] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[ 696.397435] RBP: ffff94c4862cbe80 R08: 0000000000000002 R09:
0000000000000001
[ 696.397449] R10: 0000000000100000 R11: 0000000000000397 R12:
0000000000000006
[ 696.397462] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 696.397479] ? __sched_text_end+0x1/0x1
[ 696.397489] default_idle+0x20/0x100
[ 696.397499] arch_cpu_idle+0x15/0x20
[ 696.397507] default_idle_call+0x23/0x30
[ 696.397515] do_idle+0x172/0x1f0
[ 696.397522] cpu_startup_entry+0x73/0x80
[ 696.397530] start_secondary+0x1ab/0x200
[ 696.397538] secondary_startup_64+0xa5/0xb0
[ 696.397545] Code: 89 5d b0 49 29 cc 45 01 a7 80 00 00 00 44 89 e1 48 29 c8
48 89 4d a8 49 89 87 d8 00 00 00 89 d2 48 8d 84 d6 38 03 00 00 48 8b 00 <4c> 8b
70 10 4c 8d 68 10 4d 85 f6 0f 84 f6 00 00 00 49 8d 47 30
[ 696.397584] RIP: __cgroup_bpf_run_filter_skb+0xbb/0x1e0 RSP:
ffff893fdcb83a70
[ 696.397607] ---[ end trace ec5c84424d511a6f ]---
[ 696.397616] Kernel panic - not syncing: Fatal exception in interrupt
[ 696.397876] Kernel Offset: 0xd600000 from 0xffffffff81000000 (relocation
range: 0xffffffff80000000-0xffffffffbfffffff)
We've correlated some of the other crashes, and the ASCII was a bit of a
red herring. All the others are a NULL pointer deference in the same
place, so the problem is likely OoB memory read (possibly
use-after-free) of a piece of memory which is usually zero, but not always.
It is actually the control VM's for our test farms which were impacted,
one of which was reliably crashing every 5 minutes or so, and others on
more sporadic intervals up to about a day. In all cases, reverting to
the -108 kernel has resolved the crashes.
Unfortunately, attempts to repro this off our production environment
with a packet trace aren't going quite so well. We're still experimenting.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1886668/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
