I created a workaround for this issue while its being worked on:

http://pastebin.ubuntu.com/25519820/

Create a file: /lib/systemd/system-shutdown/debug.sh and place the
contents in it.

This workaround (based on some ideas from sil2100 and slangasek for
iscsi / umount) is basically bringing the interfaces up and remounting /
so it can finally cleanup everything.

Things to note:

- It ONLY RUNS if there are iscsi leftovers
- It uses ifupdown only for networking (/etc/network/interfaces)
- It has to remount / to do networking and to run iscsid
- If it fails to bring network it will hang like before (kernel issue)
- It waits for iscsi to be logged in again (might take awhile to shutdown)
- If logout fails, it hangs again (unless the network is left configured, i 
could change it)

What is this script different then altering open-iscsi.service ?

It runs at the very end of systemd shutdown and it is very unlikely that
there are any services holding references to iscsi mounts, disallowing
them to be logged out.

Now I'll test Debian SID and check if this is found there, to open a bug in 
Debian project as well.
Before moving on into open-iscsi services - to create a cleanup unit file for 
the open-iscsi package, like this workaround - i'll dig into the kernel issue. 
I'm afraid no fix will be as good as making sure kernel let the queued I/O cmd 
go. I have also to make sure this workaround is changed to allow root iscsi to 
be logged out.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

Status in linux package in Ubuntu:
  In Progress
Status in open-iscsi package in Ubuntu:
  In Progress
Status in linux source package in Xenial:
  In Progress
Status in open-iscsi source package in Xenial:
  In Progress
Status in linux source package in Zesty:
  In Progress
Status in open-iscsi source package in Zesty:
  In Progress
Status in linux source package in Artful:
  In Progress
Status in open-iscsi source package in Artful:
  In Progress

Bug description:
  I have 4 servers running the latest 16.04 updates from the development
  branch (as of right now).

  Each server is connected to NetApp storage using iscsi software
  initiator.  There are a total of 56 volumes spread across two NetApp
  arrays.  Each volume has 4 paths available to it which are being
  managed by device mapper.

  While logged into the iscsi sessions all I have to do is reboot the
  server and I get a hang.

  I see a message that says:

    "Reached target Shutdown"

  followed by

    "systemd-shutdown[1]: Failed to finalize DM devices, ignoring"

  and then I see 8 lines that say:

    "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 
4311815***, last ping 43118164**, now 4311817***"
    NOTE: the actual values of the *'s differ for each line above.

  This seems like a bug somewhere but I am unaware of any additional
  logging that I could turn on to pinpoint the problem.

  Note I also have similar setups that are not doing iscsi and they
  don't have this problem.

  Here is a screenshot of what I see on the shell when I try to reboot:

  (https://launchpadlibrarian.net/291303059/Screenshot.jpg)

  This is being tracked in NetApp bug tracker CQ number 860251.

  If I log out of all iscsi sessions before rebooting then I do not
  experience the hang:

  iscsiadm -m node -U all

  We are wondering if this could be some kind of shutdown ordering
  problem.  Like the network devices have already disappeared and then
  iscsi tries to perform some operation (hence the ping timeouts).

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to