I uploaded a new debug kernel to the ppa: https://launchpad.net/~ddstreet/+archive/ubuntu/lp1711407
I've only updated the 4.13 kernel currently, on Artful, so if you need a Xenial HWE 4.13 kernel let me know and I can build that as well. It will likely take hours to finish building but should be ready tomorrow. With the debug kernel, when you hit the bug, you should see additional debug output, like: [ 87.028053] unregister_netdevice: waiting for lo (netns ffff9af471fa8000) to become free. Usage count = 1 [ 87.031461] (netns ffff9af471fa8000): open sk ffff9af479983000 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 [ 87.031464] (netns ffff9af471fa8000): open sk ffff9af479982000 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 [ 87.031467] (netns ffff9af471fa8000): open sk ffff9af479982800 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 [ 87.031470] (netns ffff9af471fa8000): open sk ffff9af479985000 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 [ 87.031475] (netns ffff9af471fa8000): open sk ffff9af476541540 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031479] (netns ffff9af471fa8000): open sk ffff9af471096940 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031482] (netns ffff9af471fa8000): open sk ffff9af471094f00 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031485] (netns ffff9af471fa8000): open sk ffff9af4710970c0 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031488] (netns ffff9af471fa8000): open sk ffff9af471094000 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031491] (netns ffff9af471fa8000): open sk ffff9af4710961c0 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031494] (netns ffff9af471fa8000): open sk ffff9af471095680 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031497] (netns ffff9af471fa8000): open sk ffff9af471096580 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031501] (netns ffff9af471fa8000): open sk ffff9af4710943c0 family 2 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031504] (netns ffff9af471fa8000): open sk ffff9af479987000 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 [ 87.031507] (netns ffff9af471fa8000): open sk ffff9af479981800 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 [ 87.031510] (netns ffff9af471fa8000): open sk ffff9af475703f00 family 10 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031513] (netns ffff9af471fa8000): open sk ffff9af475704380 family 10 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031516] (netns ffff9af471fa8000): open sk ffff9af475707080 family 10 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031520] (netns ffff9af471fa8000): open sk ffff9af475705580 family 10 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031523] (netns ffff9af471fa8000): open sk ffff9af475703a80 family 10 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031526] (netns ffff9af471fa8000): open sk ffff9af475702400 family 10 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031529] (netns ffff9af471fa8000): open sk ffff9af475701f80 family 10 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031532] (netns ffff9af471fa8000): open sk ffff9af475702880 family 10 state 7 dst_cache (null) rx_dst (null) creator inet_ctl_sock_create+0x37/0x80 [ 87.031535] (netns ffff9af471fa8000): open sk ffff9af471fc0800 family 16 state 7 dst_cache (null) rx_dst (null) creator __netlink_kernel_create+0x6c/0x260 [ 87.031567] (netns ffff9af471fa8000): open sk ffff9af473ffa100 family 2 state 4 dst_cache (null) rx_dst ffff9af471ec4a00 creator generic_ip_connect+0x24b/0x4f0 [cifs] That will include all open kernel sockets for the netns that's exiting, with dbg for the sockets. Assuming the problem is caused by a kernel socket remaining open (with dst(s) holding reference(s) to the lo interface), we should be able to tell where the socket was created and what type of socket it is. Of course, if the problem is a simple dst leak, this won't help show where it came from, but the debug output should still help in that case, to at least show the issue isn't being caused by a kernel socket. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1711407 Title: unregister_netdevice: waiting for lo to become free Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in linux source package in Bionic: In Progress Bug description: This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug. -- [Impact] When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs: unregister_netdevice: waiting for lo to become free. Usage count = 1 This can cause issues when trying to create net network namespace and thus block a user from creating new containers. [Test Case] See comment 16, reproducer provided at https://github.com/fho/docker- samba-loop To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1711407/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
