On Thu, May 28, 2015 at 10:41 PM, Andrey Korolyov <and...@xdel.ru> wrote: > Hi, > > I am currently playing with SYNPROXY target to optimize SYN filtering > performance and by occasion found that TCP SYN packets containing port > 0 can result in a soft lockup when conntrack is enabled just by > itself, given high packet ratio (I`ve reached 450kpps so far with 60b > packets on a /32<->/32 flood with enabled flow control at the media > level and middle-level E3 Xeon on receiver side). Same flood with port >> 0 going just well, producing same ceil numbers but without visible > lockups in kernel log. I`ve tested the issue on a broad range of 3.x > kernels and all of them are seemingly affected. Fast and dirty grep > revealed special conditions for port 0 only for protocol-specific > helpers, but there are none of them. > > Please find both same captures and traceback below.
Attached trace without GSO, at its presence can be somehow confusing in a previous sample. The testbed using net.nf_conntrack_max = 2000000, forgot to mention that previously.
[52671.706307] BUG: soft lockup - CPU#0 stuck for 24s! [rcuos/0:18] [52671.706331] Modules linked in: ixgbe mdio xt_CT iptable_raw ipt_SYNPROXY nf_synproxy_core nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp xt_conntrack nf_conntrack iptable_filter ip_tables x_tables tun openvswitch nfsd auth_rpcgss oid_registry nfs_acl nfs lockd dns_resolver fscache sunrpc bridge stp llc w83627ehf hwmon_vid loop fuse dm_crypt dm_mod coretemp kvm_intel snd_pcm kvm snd_page_alloc snd_timer crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 ablk_helper snd cryptd lrw iTCO_wdt soundcore iTCO_vendor_support gf128mul glue_helper joydev evdev pcspkr video processor button i2c_i801 lpc_ich mfd_core shpchp ext4 crc16 jbd2 mbcache microcode sg sd_mod crc_t10dif hid_generic usbhid hid ahci libahci libata mpt2sas igb raid_class i2c_algo_bit dca scsi_transport_sas ptp pps_core i2c_core ehci_pci [52671.706364] scsi_mod ehci_hcd xhci_hcd usbcore usb_common thermal fan thermal_sys [52671.706369] CPU: 0 PID: 18 Comm: rcuos/0 Not tainted 3.10-0.bpo.3-amd64 #1 Debian 3.10.11-1~bpo70+19 [52671.706370] Hardware name: Supermicro X10SL7-F/X10SL7-F, BIOS 2.00 04/24/2014 [52671.706372] task: ffff88040d528790 ti: ffff88040d532000 task.ti: ffff88040d532000 [52671.706373] RIP: 0010:[<ffffffff812bf927>] [<ffffffff812bf927>] sock_wfree+0x42/0x5b [52671.706377] RSP: 0018:ffff88041fc03b18 EFLAGS: 00000202 [52671.706378] RAX: ffffffff812bf801 RBX: ffff88041fc10110 RCX: 0000000000000024 [52671.706379] RDX: ffff8802e0751610 RSI: 00000000ffffffff RDI: ffff88041fc0fe40 [52671.706380] RBP: 0000000000000300 R08: 0000000000000000 R09: 0000000000000000 [52671.706381] R10: ffff88041fc0ff44 R11: 0000000000000001 R12: ffff88041fc03a88 [52671.706382] R13: ffffffff8139705d R14: 0000000000000300 R15: ffff88041fc0fe40 [52671.706383] FS: 0000000000000000(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000 [52671.706384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [52671.706385] CR2: 00007fb866cc2140 CR3: 000000000160c000 CR4: 00000000001407f0 [52671.706386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [52671.706387] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [52671.706388] Stack: [52671.706389] ffff88040ac7b000 ffff88040ac7aa00 ffff88040ac7b000 ffffffff812f8747 [52671.706391] ffff88041fc0fe40 ffffffff812fb0d8 ffff88041fc03b80 ffff88041fc03b78 [52671.706392] ffffffff00000040 0000000000000286 ffffffff817dda00 ffffffff0214140a [52671.706394] Call Trace: [52671.706395] <IRQ> [52671.706399] [<ffffffff812f8747>] ? skb_orphan+0x12/0x27 [52671.706402] [<ffffffff812fb0d8>] ? ip_send_unicast_reply+0x243/0x297 [52671.706406] [<ffffffff810477bf>] ? mod_timer+0x7b/0x89 [52671.706409] [<ffffffff813120c2>] ? tcp_v4_send_reset+0x2db/0x324 [52671.706411] [<ffffffff81312e56>] ? tcp_v4_rcv+0x387/0x559 [52671.706413] [<ffffffff812f6017>] ? __xfrm_policy_check2.constprop.9+0x50/0x50 [52671.706415] [<ffffffff812f6117>] ? ip_local_deliver_finish+0x100/0x176 [52671.706418] [<ffffffff812cd640>] ? __netif_receive_skb_core+0x447/0x4bf [52671.706420] [<ffffffff812cd893>] ? netif_receive_skb+0x4c/0x7d [52671.706422] [<ffffffff812ce013>] ? napi_gro_receive+0x35/0x76 [52671.706427] [<ffffffffa04ab24c>] ? ixgbe_poll+0xbc9/0xe0a [ixgbe] [52671.706429] [<ffffffff812cddaa>] ? net_rx_action+0xa7/0x1e1 [52671.706431] [<ffffffff8106442c>] ? account_system_time+0x113/0x12c [52671.706433] [<ffffffff81041683>] ? __do_softirq+0xf1/0x216 [52671.706436] [<ffffffff8139781c>] ? call_softirq+0x1c/0x30 [52671.706436] <EOI> [52671.706439] [<ffffffff8100eade>] ? do_softirq+0x3a/0x78 [52671.706440] [<ffffffff8104142e>] ? _local_bh_enable_ip.isra.11+0x6a/0x88 [52671.706443] [<ffffffff810a2413>] ? rcu_nocb_kthread+0x25e/0x298 [52671.706445] [<ffffffff810573e3>] ? abort_exclusive_wait+0x79/0x79 [52671.706447] [<ffffffff810a21b5>] ? force_qs_rnp+0x120/0x120 [52671.706448] [<ffffffff810a21b5>] ? force_qs_rnp+0x120/0x120 [52671.706450] [<ffffffff81056a54>] ? kthread+0x81/0x89 [52671.706452] [<ffffffff810126f9>] ? paravirt_read_tsc+0x5/0x8 [52671.706454] [<ffffffff810569d3>] ? __kthread_parkme+0x5d/0x5d [52671.706457] [<ffffffff813963bc>] ? ret_from_fork+0x7c/0xb0 [52671.706458] [<ffffffff810569d3>] ? __kthread_parkme+0x5d/0x5d [52671.706459] Code: e6 ff ff 84 c0 75 1d 8d 7d ff bd 01 00 00 00 48 8d b3 04 01 00 00 e8 7d e3 ff ff 48 89 df ff 93 70 02 00 00 f0 29 ab 04 01 00 00 <40> 0f 94 c5 40 84 ed 74 0b 48 89 df 5b 5b 5d e9 94 fe ff ff 41
