Hypothesis,
Test (2) - The error is propagated to upper layers after X seconds.
In this case I'm testing:
$ sudo iscsiadm -m node -o show | grep node.session.timeo.replac
node.session.timeo.replacement_timeout = 60
node.session.timeo.replacement_timeout = 60
And same "locking" behavior is observed, but for a certain period of
time. Now, shutdown procedure stayed locked for exact 60 seconds before
allowing the machine to shutdown. This means that the systemd logic
NEEDS either a clear shutdown, for the _netdev filesystems, OR the error
to be propagated so the umount() can go on:
[ 17.612020] connection1:0: ping timeout of 5 secs expired, recv timeout 5,
last rx 4294894197, last ping 4294895448, now 4294896700
[ 17.644128] connection2:0: ping timeout of 5 secs expired, recv timeout 5,
last rx 4294894204, last ping 4294895456, now 4294896708
<60 seconds>
[ 78.044264] sd 2:0:0:1: rejecting I/O to offline device
[ 78.045297] blk_update_request: I/O error, dev sdb, sector 0
[ 78.046315] sd 3:0:0:1: rejecting I/O to offline device
[ 78.046357] sd 2:0:0:1: rejecting I/O to offline device
[ 78.046363] blk_update_request: I/O error, dev sdb, sector 0
[ 78.046487] sd 2:0:0:1: rejecting I/O to offline device
[ 78.050117] blk_update_request: I/O error, dev sda, sector 20481074
[ 78.051219] XFS (sda1): metadata I/O error: block 0x1387c32 ("xlog_iodone")
error 5 numblks 64
[ 78.052727] XFS (sda1): Log I/O Error Detected. Shutting down filesystem
[ 78.053901] XFS (sda1): Please umount the filesystem and rectify the
problem(s)
[ 78.060723] sd 3:0:0:1: rejecting I/O to offline device
[ 78.061759] blk_update_request: I/O error, dev sda, sector 0
<error was propagated>
<filesystems can proceed with umount>
[ OK ] Unmounted /ext4.
[ OK ] Unmounted /xfs.
[ OK ] Stopped File System Check on /dev/disk/by-label/XFS.
[ OK ] Stopped File System Check on /dev/disk/by-label/EXT4.
[ OK ] Removed slice system-systemd\x2dfsck.slice.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925
Title:
Shutdown hang on 16.04 with iscsi targets
Status in systemd package in Ubuntu:
Confirmed
Status in systemd source package in Xenial:
In Progress
Bug description:
I have 4 servers running the latest 16.04 updates from the development
branch (as of right now).
Each server is connected to NetApp storage using iscsi software
initiator. There are a total of 56 volumes spread across two NetApp
arrays. Each volume has 4 paths available to it which are being
managed by device mapper.
While logged into the iscsi sessions all I have to do is reboot the
server and I get a hang.
I see a message that says:
"Reached target Shutdown"
followed by
"systemd-shutdown[1]: Failed to finalize DM devices, ignoring"
and then I see 8 lines that say:
"connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx
4311815***, last ping 43118164**, now 4311817***"
"connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx
4311815***, last ping 43118164**, now 4311817***"
"connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx
4311815***, last ping 43118164**, now 4311817***"
"connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx
4311815***, last ping 43118164**, now 4311817***"
"connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx
4311815***, last ping 43118164**, now 4311817***"
"connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx
4311815***, last ping 43118164**, now 4311817***"
"connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx
4311815***, last ping 43118164**, now 4311817***"
"connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx
4311815***, last ping 43118164**, now 4311817***"
NOTE: the actual values of the *'s differ for each line above.
This seems like a bug somewhere but I am unaware of any additional
logging that I could turn on to pinpoint the problem.
Note I also have similar setups that are not doing iscsi and they
don't have this problem.
Here is a screenshot of what I see on the shell when I try to reboot:
(https://launchpadlibrarian.net/291303059/Screenshot.jpg)
This is being tracked in NetApp bug tracker CQ number 860251.
If I log out of all iscsi sessions before rebooting then I do not
experience the hang:
iscsiadm -m node -U all
We are wondering if this could be some kind of shutdown ordering
problem. Like the network devices have already disappeared and then
iscsi tries to perform some operation (hence the ping timeouts).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
