Begin forwarded message:
Date: Tue, 8 Dec 2015 13:26:51 +0000 From: "bugzilla-dae...@bugzilla.kernel.org" <bugzilla-dae...@bugzilla.kernel.org> To: "shemmin...@linux-foundation.org" <shemmin...@linux-foundation.org> Subject: [Bug 109071] New: Kernel bug in skbuff.c: BUG_ON(len) crashes in combination with IPv6 and GRE tunnels https://bugzilla.kernel.org/show_bug.cgi?id=109071 Bug ID: 109071 Summary: Kernel bug in skbuff.c: BUG_ON(len) crashes in combination with IPv6 and GRE tunnels Product: Networking Version: 2.5 Kernel Version: Linux skjennungen.simula.nornet 4.2.0-19-generic #23~14.04.1-Ubuntu SMP Thu Nov 12 12:33:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Hardware: All OS: Linux Tree: Mainline Status: NEW Severity: high Priority: P1 Component: Other Assignee: shemmin...@linux-foundation.org Reporter: dre...@iem.uni-due.de Regression: No I use a machine with kernel 4.2.0 (64 bit, Ubuntu 14.04) that has 6 Ethernet interfaces. The machine is running von KVM VM, five of the interfaces are bridged into the VM. The VM also runs Ubuntu 14.04, 64 bit with the same kernel. Over the virtual interfaces, the VM creates GRE tunnels transporting IPv4 and IPv6 packets. GRE is used with key, but without checksum => MTU is 1472 bytes. The following kernel crashes happen quite frequently with several machines: [ 1881.204653] kernel BUG at /build/linux-lts-wily-1zclH3/linux-lts-wily-4.2.0/net/core/skbuff.c:2097! [ 1881.204784] invalid opcode: 0000 [#1] SMP [ 1881.204853] Modules linked in: vhost_net vhost macvtap macvlan ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_CHECKSUM iptable_mangle xt_tcpudp ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables autofs4 deflate ctr twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic camellia_x86_64 serpent_sse2_x86_64 xts serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic cmac xcbc rmd160 crypto_null af_key xfrm_algo nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc fscache intel_powerclamp coretemp kvm_intel gpio_ich amdkfd amd_iommu_v2 kvm radeon ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel [ 1881.206234] aes_x86_64 bridge input_leds lrw stp joydev gf128mul llc glue_helper ablk_helper ttm cryptd drm_kms_helper drm serio_raw i2c_algo_bit hpilo ipmi_si lpc_ich ipmi_msghandler i7core_edac 8250_fintek mac_hid edac_core shpchp dummy lp parport reiserfs hid_generic psmouse pata_acpi usbhid hid tg3 e1000e ptp pps_core [ 1881.206839] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G I 4.2.0-19-generic #23~14.04.1-Ubuntu [ 1881.206975] Hardware name: HP ProLiant DL320 G6, BIOS W07 07/02/2013 [ 1881.207066] task: ffff88020e29b200 ti: ffff88020e2ac000 task.ti: ffff88020e2ac000 [ 1881.207173] RIP: 0010:[<ffffffff8169eb59>] [<ffffffff8169eb59>] __skb_checksum+0x2c9/0x2d0 [ 1881.207304] RSP: 0018:ffff8802174c35e8 EFLAGS: 00010286 [ 1881.207381] RAX: ffff8800c2a86840 RBX: 00000000fffef742 RCX: ffff8801eff52000 [ 1881.207483] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88020e29b200 [ 1881.207593] RBP: ffff8802174c3658 R08: ffff8802174c3668 R09: 0000000000000000 [ 1881.207725] R10: 0000000000000000 R11: 0000000000000040 R12: 0000000000000000 [ 1881.207826] R13: 00000000fffef742 R14: 00000000fffef742 R15: 0000000000000001 [ 1881.207928] FS: 0000000000000000(0000) GS:ffff8802174c0000(0000) knlGS:0000000000000000 [ 1881.208045] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1881.208126] CR2: 00000000028d3000 CR3: 0000000001c0d000 CR4: 00000000000026e0 [ 1881.208227] Stack: [ 1881.208257] ffff8802174d6640 0000000000000046 ffff88020e29b200 ffff88020e29b200 [ 1881.208378] ffff8800fffef742 ffff8801eff52000 0000000000000000 ffff8802174c3668 [ 1881.208500] ffff8802174c3648 ffff8801eff52000 0000000000000001 ffff8800c2a87660 [ 1881.208622] Call Trace: [ 1881.208660] <IRQ> [ 1881.208712] [<ffffffff8169eb86>] skb_checksum+0x26/0x30 [ 1881.208800] [<ffffffff8169bc10>] ? skb_push+0x40/0x40 [ 1881.208878] [<ffffffff8169b920>] ? reqsk_fastopen_remove+0x160/0x160 [ 1881.208971] [<ffffffff8178dfda>] udp6_ufo_fragment+0xba/0x2e0 [ 1881.209061] [<ffffffff810b6bf5>] ? __wake_up_common+0x55/0x90 [ 1881.209147] [<ffffffff8178d9d9>] ipv6_gso_segment+0x109/0x2a0 [ 1881.209235] [<ffffffff816b1eb5>] skb_mac_gso_segment+0x95/0xf0 [ 1881.209324] [<ffffffff81733f57>] gre_gso_segment+0x167/0x440 [ 1881.209408] [<ffffffff817b013c>] ? __slab_free+0x104/0x25c [ 1881.209491] [<ffffffff817249d3>] inet_gso_segment+0x163/0x360 [ 1881.209578] [<ffffffff816b1eb5>] skb_mac_gso_segment+0x95/0xf0 [ 1881.209665] [<ffffffff816b1f73>] __skb_gso_segment+0x63/0x90 [ 1881.209747] [<ffffffff816b22a3>] validate_xmit_skb.isra.101.part.102+0x123/0x2b0 [ 1881.209894] [<ffffffff816b280f>] validate_xmit_skb_list+0x3f/0x60 [ 1881.214432] [<ffffffff816d451d>] sch_direct_xmit+0xcd/0x1e0 [ 1881.218910] [<ffffffffc06f029a>] ? ebt_do_table+0x55a/0x64c [ebtables] [ 1881.223445] [<ffffffff816d46c3>] __qdisc_run+0x93/0x1b0 [ 1881.227887] [<ffffffff816b2bec>] __dev_queue_xmit+0x2cc/0x550 [ 1881.232311] [<ffffffff816b2e83>] dev_queue_xmit_sk+0x13/0x20 [ 1881.236629] [<ffffffffc02acbd5>] br_dev_queue_push_xmit+0x125/0x170 [bridge] [ 1881.240936] [<ffffffffc02acd7a>] br_forward_finish+0x2a/0x80 [bridge] [ 1881.245106] [<ffffffff813b21d1>] ? csum_partial+0x11/0x20 [ 1881.249207] [<ffffffffc02acab0>] ? deliver_clone+0x60/0x60 [bridge] [ 1881.253138] [<ffffffffc02ace58>] __br_forward+0x88/0x110 [bridge] [ 1881.257012] [<ffffffffc02ad287>] br_forward+0x87/0xa0 [bridge] [ 1881.260788] [<ffffffffc02ae135>] br_handle_frame_finish+0x145/0x580 [bridge] [ 1881.264474] [<ffffffffc06d704a>] ? ebt_nat_in+0x2a/0x30 [ebtable_nat] [ 1881.268071] [<ffffffff816e3c51>] ? nf_iterate+0x51/0x80 [ 1881.271503] [<ffffffff816e3ceb>] ? nf_hook_slow+0x6b/0xc0 [ 1881.274842] [<ffffffffc02ae6b6>] br_handle_frame+0x146/0x270 [bridge] [ 1881.278191] [<ffffffffc02adff0>] ? br_handle_local_finish+0x80/0x80 [bridge] [ 1881.281467] [<ffffffff816b0182>] __netif_receive_skb_core+0x1d2/0x9a0 [ 1881.284760] [<ffffffffc0065935>] ? e1000_alloc_rx_buffers+0x75/0x240 [e1000e] [ 1881.287985] [<ffffffff816b0968>] __netif_receive_skb+0x18/0x60 [ 1881.291180] [<ffffffff816b09d3>] netif_receive_skb_internal+0x23/0x80 [ 1881.294432] [<ffffffff816b0b28>] napi_gro_complete+0x98/0xd0 [ 1881.297602] [<ffffffff816b0bc3>] napi_gro_flush+0x63/0x90 [ 1881.300746] [<ffffffff816b0c57>] napi_complete_done+0x67/0xa0 [ 1881.303959] [<ffffffffc006a25a>] e1000e_poll+0xba/0x2a0 [e1000e] [ 1881.307091] [<ffffffff817bd5fa>] ? do_IRQ+0x5a/0xe0 [ 1881.310220] [<ffffffff816b0ddc>] net_rx_action+0x14c/0x320 [ 1881.313279] [<ffffffff8107b3d2>] __do_softirq+0xd2/0x250 [ 1881.316251] [<ffffffff8107b785>] irq_exit+0x95/0xa0 [ 1881.319260] [<ffffffff817bd5fa>] do_IRQ+0x5a/0xe0 [ 1881.322196] [<ffffffff817bb56b>] common_interrupt+0x6b/0x6b [ 1881.325155] <EOI> [ 1881.325189] [<ffffffff810ef0a8>] ? tick_program_event+0x48/0x80 [ 1881.331038] [<ffffffff81654c45>] ? cpuidle_enter_state+0xb5/0x220 [ 1881.334028] [<ffffffff81654c24>] ? cpuidle_enter_state+0x94/0x220 [ 1881.337006] [<ffffffff81654de7>] cpuidle_enter+0x17/0x20 [ 1881.339936] [<ffffffff810b76eb>] call_cpuidle+0x3b/0x70 [ 1881.342872] [<ffffffff81654dc3>] ? cpuidle_select+0x13/0x20 [ 1881.345855] [<ffffffff810b798c>] cpu_startup_entry+0x26c/0x330 [ 1881.348761] [<ffffffff8104b1a5>] start_secondary+0x175/0x1a0 [ 1881.351699] Code: e8 2d 89 9d ff 8b 45 9c e9 ab fe ff ff be 20 08 00 00 48 c7 c7 c0 78 b4 81 44 89 55 c0 e8 10 89 9d ff 44 8b 55 c0 e9 05 ff ff ff <0f> 0b 0f 1f 44 00 00 66 66 66 66 90 55 48 89 e5 48 83 ec 10 4c [ 1881.358334] RIP [<ffffffff8169eb59>] __skb_checksum+0x2c9/0x2d0 [ 1881.361567] RSP <ffff8802174c35e8> Line 2097 in skbuff.c is: BUG_ON(len); The problem seems to be somewhere in receive offloading, related to IPv6 and the tunnels, and possibly related to KVM. I also noticed similar (possibly the same) issue when using VirtualBox instead of KVM (see https://www.virtualbox.org/ticket/14779 with the VirtualBox bug report). If necessary, I can provide plenty of Kdump-generated kernel dumps. -- You are receiving this mail because: You are the assignee for the bug. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
