This is what I discovered: This systemd issue is caused because there are iscsi connections still logged in (not because of multipath, bonding, vlans, etc). One way of reproducing this is by simply editing the open-iscsi.service unit and disabling the logout script. It will, for me in all times, cause the same hang as we are talking to.
inaddy@iscsihang:~$ sudo systemctl edit --full open-iscsi.service ... ExecStop=/lib/open-iscsi/umountiscsi.sh ExecStop=/bin/sync #ExecStop=/lib/open-iscsi/logout-all.sh But of course no one is doing this. So, reading logout-all you will discover that it logs out all paths but it has some caveats: ISCSI_ROOT_KEEP_ALL_SESSIONS_AT_SHUTDOWN (from /etc/default/open-iscsi) and /run/open-iscsi/shutdown-keep-sessions would cause those iscsi sessions not to be logged out. Still, I wasn't using neither to reproduce so I found out something else. Disabling both systemd services: inaddy@iscsihang:~$ systemctl list-unit-files --full | grep iscsi iscsi.service disabled (alias for iscsid) iscsid.service disabled open-iscsi.service disabled I was still able to reproduce the issue: http://pastebin.ubuntu.com/25388348/ By issuing a logout when the iscsid daemon is gone. It looks like iscsid is lost after it has died/been killed on the sessions that were previously established. If you try to login again, it will be totally lost when trying to logout, causing leftovers (opened iscsi sessions) causing systemd to hang (even not having daemons running, in this case). I'm investigating: - Should systemd hang on those leftovers ? Since there are caveats that explicitly allow you to leave opened sessions, for / for example, I doubt it. - Does iscsid daemon contain a bug that causes sessions to remain un- mapped and opened if daemon is restarted without a logout of previously opened sessions ? Now that I'm able to reproduce this at will, I'll be able to answer those questions soon. Best, Rafael PS: Matt, I asked partnership program managers to reach out to understand better your future needs. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Xenial: In Progress Bug description: I have 4 servers running the latest 16.04 updates from the development branch (as of right now). Each server is connected to NetApp storage using iscsi software initiator. There are a total of 56 volumes spread across two NetApp arrays. Each volume has 4 paths available to it which are being managed by device mapper. While logged into the iscsi sessions all I have to do is reboot the server and I get a hang. I see a message that says: "Reached target Shutdown" followed by "systemd-shutdown[1]: Failed to finalize DM devices, ignoring" and then I see 8 lines that say: "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" NOTE: the actual values of the *'s differ for each line above. This seems like a bug somewhere but I am unaware of any additional logging that I could turn on to pinpoint the problem. Note I also have similar setups that are not doing iscsi and they don't have this problem. Here is a screenshot of what I see on the shell when I try to reboot: (https://launchpadlibrarian.net/291303059/Screenshot.jpg) This is being tracked in NetApp bug tracker CQ number 860251. If I log out of all iscsi sessions before rebooting then I do not experience the hang: iscsiadm -m node -U all We are wondering if this could be some kind of shutdown ordering problem. Like the network devices have already disappeared and then iscsi tries to perform some operation (hence the ping timeouts). To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
