Hello Matt, Good to read your feedback. When you have multipath AND iscsi, the error propagation timeout can be given by one or both, that is why I explained both. This issue is not an iscsi or multipath problem, it is a systemd problem related to the fact that the network is removed before the disks are unmounted. Good to know you don't have containers, that is something that could hold the disk superblock reference as well (even not having containers, filesystems mounted by _netdev can hold reference by different mount namespaces started by systemd itself, not the case here).
Could you do me a favor. Are you reproducing this issue in a constant basis ? If you are, could you please reduce your replacement_timeout to 0 and see if you can reproduce it again ? Please be aware that, after changing replacement_timeout = 0 in iscsid.conf, you will have to change the parameter for the disks already discovered: ## example Change iscsid.conf AND: $ sudo iscsiadm -m node 192.168.48.1:3260,1 iqn.2017.tgtd:tid2.lun 192.168.48.1:3260,1 iqn.2017.tgtd:tid1.lun $ sudo iscsiadm -m node -o show | grep node.session.timeo.replacement_timeout node.session.timeo.replacement_timeout = 20 node.session.timeo.replacement_timeout = 20 $ sudo iscsiadm -m node -o update -n node.session.timeo.replacement_timeout -v 0 $ sudo iscsiadm -m node -o show | grep node.session.timeo.replacement_timeout node.session.timeo.replacement_timeout = 0 node.session.timeo.replacement_timeout = 0 This will cause the I/O errors to be propagated right away - when systemd disconnects the interface - and cause the later umount - done by other systemd service - to succeed without waiting for pagecache timeout + iscsi timeout (20 for you). Hopefully the "0" will be fast enough to make systemd to continue all the times. Could you please let me know if it mitigates the issue ? If it does, I'll work in the actual fix. This will just confirm hypothesis and help in diagnose (and serve as a workaround if anyone cares). Thank you -Rafael -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Xenial: In Progress Bug description: I have 4 servers running the latest 16.04 updates from the development branch (as of right now). Each server is connected to NetApp storage using iscsi software initiator. There are a total of 56 volumes spread across two NetApp arrays. Each volume has 4 paths available to it which are being managed by device mapper. While logged into the iscsi sessions all I have to do is reboot the server and I get a hang. I see a message that says: "Reached target Shutdown" followed by "systemd-shutdown[1]: Failed to finalize DM devices, ignoring" and then I see 8 lines that say: "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" NOTE: the actual values of the *'s differ for each line above. This seems like a bug somewhere but I am unaware of any additional logging that I could turn on to pinpoint the problem. Note I also have similar setups that are not doing iscsi and they don't have this problem. Here is a screenshot of what I see on the shell when I try to reboot: (https://launchpadlibrarian.net/291303059/Screenshot.jpg) This is being tracked in NetApp bug tracker CQ number 860251. If I log out of all iscsi sessions before rebooting then I do not experience the hang: iscsiadm -m node -U all We are wondering if this could be some kind of shutdown ordering problem. Like the network devices have already disappeared and then iscsi tries to perform some operation (hence the ping timeouts). To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
