On Wed, Nov 18, 2015 at 2:11 AM, Arkadiusz Miskiewicz <a.miskiew...@gmail.com> wrote: > > Hi. > > I'm running 4.3.0 kernel (+ few backported mm patches). Machine is doing many > rsnapshots > in parallel (so heavily loaded). > > I'm seeing this from time to time (below). What would be the reason for so > long hangs?
It is not a lockdep problem, it is that someone was holding that mutex for a long time or that mutex was not released correctly. Can you do a sysrq-t when you reproduce this problem? Thanks! > > [ 2278.385668] INFO: task snmpd:1775 blocked for more than 120 seconds. > [ 2278.392089] Not tainted 4.3.0-0.1 #1 > [ 2278.396226] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [ 2278.404111] snmpd D ffff88081fc158c0 0 1775 1 > 0x00000000 > [ 2278.411229] ffff8807e6a8bcf8 0000000000000082 ffffffff81c13500 > ffff8807ebb88d80 > [ 2278.418753] ffff8807e6a8c000 ffff8807ebb88d80 ffffffff81cc7c84 > 00000000ffffffff > [ 2278.426286] ffffffff81cc7c88 ffff8807e6a8bd10 ffffffff8161c203 > ffffffff81cc7c80 > [ 2278.433854] Call Trace: > [ 2278.436341] [<ffffffff8161c203>] schedule+0x33/0x80 > [ 2278.441349] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10 > [ 2278.447733] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120 > [ 2278.453956] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30 > [ 2278.459133] [<ffffffff81523f15>] rtnl_lock+0x15/0x20 > [ 2278.464212] [<ffffffff8158b902>] devinet_ioctl+0xd2/0x6e0 > [ 2278.469757] [<ffffffff81208c14>] ? mntput+0x24/0x40 > [ 2278.474769] [<ffffffff811ebe62>] ? __fput+0x142/0x200 > [ 2278.479955] [<ffffffff8158d5a0>] inet_ioctl+0x50/0x70 > [ 2278.485122] [<ffffffff814f6e05>] sock_do_ioctl+0x25/0x50 > [ 2278.490578] [<ffffffff814f7322>] sock_ioctl+0x1d2/0x290 > [ 2278.495920] [<ffffffff811fc3f5>] do_vfs_ioctl+0x295/0x470 > [ 2278.501444] [<ffffffff814f6cd6>] ? sock_alloc_file+0xa6/0x130 > [ 2278.507338] [<ffffffff81090eb4>] ? task_work_add+0x44/0x60 > [ 2278.512942] [<ffffffff811ebff7>] ? fput+0x47/0x90 > [ 2278.517771] [<ffffffff811fc649>] SyS_ioctl+0x79/0x90 > [ 2278.522845] [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75 > [ 2278.529341] INFO: task sudo:4534 blocked for more than 120 seconds. > [ 2278.535635] Not tainted 4.3.0-0.1 #1 > [ 2278.539759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [ 2278.547680] sudo D 0000000000000000 0 4534 1 > 0x00000004 > [ 2278.554825] ffff880080677bb8 0000000000000086 ffff8807f3874380 > ffff8807f3eb8000 > [ 2278.562389] ffff880080678000 ffff8807f3eb8000 ffffffff81cc7c84 > 00000000ffffffff > [ 2278.569914] ffffffff81cc7c88 ffff880080677bd0 ffffffff8161c203 > ffffffff81cc7c80 > [ 2278.577436] Call Trace: > [ 2278.579911] [<ffffffff8161c203>] schedule+0x33/0x80 > [ 2278.584918] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10 > [ 2278.591329] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120 > [ 2278.597571] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30 > [ 2278.602743] [<ffffffff8154746e>] __netlink_dump_start+0x5e/0x190 > [ 2278.608890] [<ffffffff81526100>] ? rtnl_fill_ifinfo+0x13f0/0x13f0 > [ 2278.615105] [<ffffffff81526eac>] rtnetlink_rcv_msg+0x1bc/0x230 > [ 2278.621054] [<ffffffff81526100>] ? rtnl_fill_ifinfo+0x13f0/0x13f0 > [ 2278.627262] [<ffffffff81526cf0>] ? rtnetlink_rcv+0x30/0x30 > [ 2278.632861] [<ffffffff815498d4>] netlink_rcv_skb+0xa4/0xc0 > [ 2278.638463] [<ffffffff81526ce8>] rtnetlink_rcv+0x28/0x30 > [ 2278.643908] [<ffffffff81548fc7>] netlink_unicast+0x127/0x1a0 > [ 2278.649692] [<ffffffff81549604>] netlink_sendmsg+0x4f4/0x5f0 > [ 2278.655480] [<ffffffff814f8bd8>] sock_sendmsg+0x38/0x50 > [ 2278.660830] [<ffffffff814fa2a1>] SyS_sendto+0x101/0x150 > [ 2278.666189] [<ffffffff814f6cd6>] ? sock_alloc_file+0xa6/0x130 > [ 2278.672064] [<ffffffff81206715>] ? fd_install+0x25/0x30 > [ 2278.677407] [<ffffffff814f985f>] ? SyS_socket+0x9f/0xf0 > [ 2278.682749] [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75 > > > > > > > > > [ 7433.492510] INFO: task bird:1503 blocked for more than 120 seconds. > [ 7433.498819] Not tainted 4.3.0-0.1 #1 > [ 7433.502954] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [ 7433.510840] bird D ffff88081fd158c0 0 1503 1 > 0x00000000 > [ 7433.517966] ffff8807eb923bb8 0000000000000086 ffff8807fb711b00 > ffff8807f38a4380 > [ 7433.525493] ffff8807eb924000 ffff8807f38a4380 ffffffff81cc7c84 > 00000000ffffffff > [ 7433.533017] ffffffff81cc7c88 ffff8807eb923bd0 ffffffff8161c203 > ffffffff81cc7c80 > [ 7433.540561] Call Trace: > [ 7433.543050] [<ffffffff8161c203>] schedule+0x33/0x80 > [ 7433.548041] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10 > [ 7433.554415] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120 > [ 7433.560624] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30 > [ 7433.565810] [<ffffffff81546be1>] netlink_dump+0x21/0x2c0 > [ 7433.571236] [<ffffffff815470c9>] netlink_recvmsg+0x249/0x3d0 > [ 7433.577005] [<ffffffff814f8eeb>] sock_recvmsg+0x3b/0x50 > [ 7433.582347] [<ffffffff814f96ab>] ___sys_recvmsg+0xdb/0x1f0 > [ 7433.587952] [<ffffffff810146ec>] ? __switch_to+0x25c/0x4a0 > [ 7433.593579] [<ffffffff810a6f17>] ? update_curr+0xb7/0x130 > [ 7433.599115] [<ffffffff810a354c>] ? __enqueue_entity+0x6c/0x70 > [ 7433.604970] [<ffffffff810a44f1>] ? set_next_entity+0x71/0x840 > [ 7433.610832] [<ffffffff810146ec>] ? __switch_to+0x25c/0x4a0 > [ 7433.616431] [<ffffffff814fa9b1>] __sys_recvmsg+0x51/0x90 > [ 7433.621873] [<ffffffff814faa02>] SyS_recvmsg+0x12/0x20 > [ 7433.627125] [<ffffffff8161f876>] entry_SYSCALL_64_fastpath+0x16/0x75 > [ 7433.633642] INFO: task kworker/3:4:12895 blocked for more than 120 seconds. > [ 7433.640651] Not tainted 4.3.0-0.1 #1 > [ 7433.644797] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [ 7433.652682] kworker/3:4 D 0000000000000000 0 12895 2 > 0x00000000 > [ 7433.659794] Workqueue: ipv6_addrconf addrconf_verify_work > [ 7433.665240] ffff8803b0073d58 0000000000000046 ffff880781adb600 > ffff880736aa3600 > [ 7433.672763] ffff8803b0074000 ffff880736aa3600 ffffffff81cc7c84 > 00000000ffffffff > [ 7433.680287] ffffffff81cc7c88 ffff8803b0073d70 ffffffff8161c203 > ffffffff81cc7c80 > [ 7433.687811] Call Trace: > [ 7433.690288] [<ffffffff8161c203>] schedule+0x33/0x80 > [ 7433.695297] [<ffffffff8161c49e>] schedule_preempt_disabled+0xe/0x10 > [ 7433.701688] [<ffffffff8161d833>] __mutex_lock_slowpath+0xb3/0x120 > [ 7433.707923] [<ffffffff8161d8bf>] mutex_lock+0x1f/0x30 > [ 7433.713093] [<ffffffff81523f15>] rtnl_lock+0x15/0x20 > [ 7433.718168] [<ffffffff815c903e>] addrconf_verify_work+0xe/0x20 > [ 7433.724114] [<ffffffff8108cc6f>] process_one_work+0x14f/0x430 > [ 7433.730006] [<ffffffff8108cf9b>] worker_thread+0x4b/0x4b0 > [ 7433.735517] [<ffffffff8161bb71>] ? __schedule+0x2a1/0x900 > [ 7433.741031] [<ffffffff8108cf50>] ? process_one_work+0x430/0x430 > [ 7433.747083] [<ffffffff8108cf50>] ? process_one_work+0x430/0x430 > [ 7433.753131] [<ffffffff81092be8>] kthread+0xd8/0xf0 > [ 7433.758063] [<ffffffff81092b10>] ? kthread_worker_fn+0x170/0x170 > [ 7433.764204] [<ffffffff8161fc0f>] ret_from_fork+0x3f/0x70 > [ 7433.769657] [<ffffffff81092b10>] ? kthread_worker_fn+0x170/0x170 > > config: > http://sprunge.us/BBLN > > # ethtool -i eth0 > driver: igb > version: 5.3.0-k > firmware-version: 3.16, 0x800004d7 > bus-info: 0000:05:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: no > > # lspci -v > 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM > Controller (rev 06) > Subsystem: Super Micro Computer Inc Device 0804 > Flags: bus master, fast devsel, latency 0 > Capabilities: [e0] Vendor Specific Information: Len=0c <?> > Kernel driver in use: ie31200_edac > Kernel modules: ie31200_edac > > 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor > PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 > Capabilities: [88] Subsystem: Super Micro Computer Inc Device 0804 > Capabilities: [80] Power Management version 3 > Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- > Capabilities: [a0] Express Root Port (Slot+), MSI 00 > Capabilities: [100] Virtual Channel > Capabilities: [140] Root Complex Link > Capabilities: [d94] #19 > Kernel driver in use: pcieport > Kernel modules: shpchp > > 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor > PCI Express x8 Controller (rev 06) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 > I/O behind bridge: 0000e000-0000efff > Memory behind bridge: f7100000-f72fffff > Capabilities: [88] Subsystem: Super Micro Computer Inc Device 0804 > Capabilities: [80] Power Management version 3 > Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- > Capabilities: [a0] Express Root Port (Slot+), MSI 00 > Capabilities: [100] Virtual Channel > Capabilities: [140] Root Complex Link > Capabilities: [d94] #19 > Kernel driver in use: pcieport > Kernel modules: shpchp > > 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family > USB xHCI (rev 05) (prog-if 30 [XHCI]) > Subsystem: Super Micro Computer Inc Device 0804 > Flags: bus master, medium devsel, latency 0, IRQ 34 > Memory at f7500000 (64-bit, non-prefetchable) [size=64K] > Capabilities: [70] Power Management version 2 > Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ > Kernel driver in use: xhci_hcd > Kernel modules: xhci_pci > > 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family > USB EHCI #2 (rev 05) (prog-if 20 [EHCI]) > Subsystem: Super Micro Computer Inc Device 0804 > Flags: bus master, medium devsel, latency 0, IRQ 16 > Memory at f7513000 (32-bit, non-prefetchable) [size=1K] > Capabilities: [50] Power Management version 2 > Capabilities: [58] Debug port: BAR=1 offset=00a0 > Capabilities: [98] PCI Advanced Features > Kernel driver in use: ehci-pci > Kernel modules: ehci_pci > > 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI > Express Root Port #1 (rev d5) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=03, subordinate=04, sec-latency=0 > I/O behind bridge: 0000d000-0000dfff > Memory behind bridge: f6000000-f70fffff > Capabilities: [40] Express Root Port (Slot+), MSI 00 > Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- > Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804 > Capabilities: [a0] Power Management version 3 > Kernel driver in use: pcieport > Kernel modules: shpchp > > 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI > Express Root Port #3 (rev d5) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=05, subordinate=05, sec-latency=0 > I/O behind bridge: 0000c000-0000cfff > Memory behind bridge: f7400000-f74fffff > Capabilities: [40] Express Root Port (Slot+), MSI 00 > Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- > Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804 > Capabilities: [a0] Power Management version 3 > Kernel driver in use: pcieport > Kernel modules: shpchp > > 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI > Express Root Port #4 (rev d5) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=06, subordinate=06, sec-latency=0 > I/O behind bridge: 0000b000-0000bfff > Memory behind bridge: f7300000-f73fffff > Capabilities: [40] Express Root Port (Slot+), MSI 00 > Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- > Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0804 > Capabilities: [a0] Power Management version 3 > Kernel driver in use: pcieport > Kernel modules: shpchp > > 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family > USB EHCI #1 (rev 05) (prog-if 20 [EHCI]) > Subsystem: Super Micro Computer Inc Device 0804 > Flags: bus master, medium devsel, latency 0, IRQ 23 > Memory at f7512000 (32-bit, non-prefetchable) [size=1K] > Capabilities: [50] Power Management version 2 > Capabilities: [58] Debug port: BAR=1 offset=00a0 > Capabilities: [98] PCI Advanced Features > Kernel driver in use: ehci-pci > Kernel modules: ehci_pci > > 00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server > Essential SKU LPC Controller (rev 05) > Subsystem: Super Micro Computer Inc Device 0804 > Flags: bus master, medium devsel, latency 0 > Capabilities: [e0] Vendor Specific Information: Len=0c <?> > Kernel driver in use: lpc_ich > Kernel modules: lpc_ich > > 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset > Family 6-port SATA Controller 1 [AHCI mode] (rev 05) (prog-if 01 [AHCI 1.0]) > Subsystem: Super Micro Computer Inc Device 0804 > Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 29 > I/O ports at f050 [size=8] > I/O ports at f040 [size=4] > I/O ports at f030 [size=8] > I/O ports at f020 [size=4] > I/O ports at f000 [size=32] > Memory at f7511000 (32-bit, non-prefetchable) [size=2K] > Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- > Capabilities: [70] Power Management version 3 > Capabilities: [a8] SATA HBA v1.0 > Kernel driver in use: ahci > Kernel modules: ahci > > 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus > Controller (rev 05) > Subsystem: Super Micro Computer Inc Device 0804 > Flags: medium devsel, IRQ 18 > Memory at f7510000 (64-bit, non-prefetchable) [size=256] > I/O ports at 0580 [size=32] > Kernel driver in use: i801_smbus > Kernel modules: i2c_i801 > > 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset > Family Thermal Management Controller (rev 05) > Subsystem: Super Micro Computer Inc Device 0804 > Flags: bus master, fast devsel, latency 0, IRQ 10 > Memory at dfff00000 (64-bit, non-prefetchable) [size=4K] > Capabilities: [50] Power Management version 3 > Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- > > 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 > PCI-Express Fusion-MPT SAS-2 (rev 05) > Subsystem: Super Micro Computer Inc Device 0691 > Flags: bus master, fast devsel, latency 0 > I/O ports at e000 [size=256] > Memory at f7240000 (64-bit, non-prefetchable) [size=64K] > Memory at f7200000 (64-bit, non-prefetchable) [size=256K] > Expansion ROM at f7100000 [disabled] [size=1M] > Capabilities: [50] Power Management version 3 > Capabilities: [68] Express Endpoint, MSI 00 > Capabilities: [d0] Vital Product Data > Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+ > Capabilities: [c0] MSI-X: Enable+ Count=16 Masked- > Capabilities: [100] Advanced Error Reporting > Capabilities: [1e0] #19 > Capabilities: [1c0] Power Budgeting <?> > Capabilities: [190] #16 > Capabilities: [148] Alternative Routing-ID Interpretation (ARI) > Kernel driver in use: mpt2sas > Kernel modules: mpt2sas > > 03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev > 03) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=03, secondary=04, subordinate=04, sec-latency=0 > I/O behind bridge: 0000d000-0000dfff > Memory behind bridge: f6000000-f70fffff > Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ > Capabilities: [78] Power Management version 3 > Capabilities: [80] Express PCI-Express to PCI/PCI-X Bridge, MSI 00 > Capabilities: [a4] Subsystem: Super Micro Computer Inc Device 0804 > Capabilities: [100] Virtual Channel > Kernel modules: shpchp > > 04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics > Family (rev 30) (prog-if 00 [VGA controller]) > Subsystem: Super Micro Computer Inc Device 0804 > Flags: bus master, medium devsel, latency 0, IRQ 11 > Memory at f6000000 (32-bit, non-prefetchable) [size=16M] > Memory at f7000000 (32-bit, non-prefetchable) [size=128K] > I/O ports at d000 [size=128] > Expansion ROM at <unassigned> [disabled] > Capabilities: [40] Power Management version 3 > Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+ > > 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network > Connection (rev 03) > Subsystem: Super Micro Computer Inc Device 1533 > Flags: bus master, fast devsel, latency 0 > Memory at f7400000 (32-bit, non-prefetchable) [size=512K] > I/O ports at c000 [size=32] > Memory at f7480000 (32-bit, non-prefetchable) [size=16K] > Capabilities: [40] Power Management version 3 > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > Capabilities: [a0] Express Endpoint, MSI 00 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number 00-25-90-ff-ff-87-5d-0e > Capabilities: [1a0] Transaction Processing Hints > Kernel driver in use: igb > Kernel modules: igb > > 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network > Connection (rev 03) > Subsystem: Super Micro Computer Inc Device 1533 > Flags: bus master, fast devsel, latency 0 > Memory at f7300000 (32-bit, non-prefetchable) [size=512K] > I/O ports at b000 [size=32] > Memory at f7380000 (32-bit, non-prefetchable) [size=16K] > Capabilities: [40] Power Management version 3 > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > Capabilities: [a0] Express Endpoint, MSI 00 > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number 00-25-90-ff-ff-87-5d-0f > Capabilities: [1a0] Transaction Processing Hints > Kernel driver in use: igb > Kernel modules: igb > > > -- > Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
