On Wed, 2015-12-30 at 13:34 +0100, Christian Seiler wrote: > On 12/30/2015 12:35 PM, Ritesh Raj Sarraf wrote: > > But the change you've proposed has other repercussion. It breaks > > shutdown, because now, iscsi service/sessions are terminated way > > early. > > This leaves, effectively, a hung sanboot system. > > > > Please see attached images. > > > > I also didn't like that iscsi shutdown, even thought my rootfs was > > running. > > > > I'll keep digging. > > Ok, so I think there's two things going on here: a problem in a > change I made in that doesn't work in your specific configuration > (which we should debug) and a general question about how to > handle shutdown. > > First of all: the fact that iSCSI volumes are shut down is a > change I made on purpose. There are two parts to this: first of > all, iscsid is going to be killed anyway,[*] but the kernel will > keep running sessions open regardless (unless there's an error > with the connection / the target). Secondly, which is the change > I did make, is that iSCSI sessions are logged out on shutdown > regardless of whether the rootfs is on iSCSI. I did that because > I think it is still correct to log out of sessions that the > rootfs is NOT on, because we want to cleanly logout of anything > that is possible to log out of. There is a flag in > /etc/default/open-iscsi to return to the old behavior, this is > also mentioned in the NEWS file, which is shown on updates, but > not on new installs. Note that this does not include the rootfs > sessions - they SHOULD be excluded. (Of course, if you disagree > with this change and think ALL sessions, not just the rootfs > sessions, should be kept open by default, we could discuss this > separately.) >
This is what is happening right now. root@debian-sanboot:~# systemctl stop open-iscsi root@debian-sanboot:~# systemctl status open-iscsi ● open-iscsi.service - Login to default iSCSI targets Loaded: loaded (/lib/systemd/system/open-iscsi.service; enabled; vendor preset: enabled) Active: inactive (dead) since Sun 2016-02-07 14:45:11 IST; 7s ago Docs: man:iscsiadm(8) man:iscsid(8) Process: 971 ExecStop=/lib/open-iscsi/logout-all.sh (code=exited, status=0/SUCCESS) Process: 968 ExecStop=/bin/sync (code=exited, status=0/SUCCESS) Process: 788 ExecStop=/lib/open-iscsi/umountiscsi.sh (code=exited, status=0/SUCCESS) Process: 601 ExecStart=/lib/open-iscsi/activate-storage.sh (code=exited, status=0/SUCCESS) Process: 567 ExecStart=/sbin/iscsiadm -m node --loginall=automatic (code=exited, status=15) Process: 559 ExecStartPre=/bin/systemctl --quiet is-active iscsid.service (code=exited, status=0/SUCCESS) Main PID: 601 (code=exited, status=0/SUCCESS) Feb 07 14:45:08 debian-sanboot umountiscsi.sh[788]: ls: cannot access /dev/sdd?*: No such ...ry Feb 07 14:45:09 debian-sanboot logout-all.sh[971]: Logging out of session [sid: 1, target:...0] Feb 07 14:45:09 debian-sanboot logout-all.sh[971]: Logout of [sid: 1, target: iqn.2003-01....l. Feb 07 14:45:10 debian-sanboot logout-all.sh[971]: Logging out of session [sid: 2, target:...0] Feb 07 14:45:10 debian-sanboot logout-all.sh[971]: Logout of [sid: 2, target: iqn.2003-01....l. Feb 07 14:45:10 debian-sanboot logout-all.sh[971]: Logging out of session [sid: 3, target:...0] Feb 07 14:45:11 debian-sanboot logout-all.sh[971]: Logout of [sid: 3, target: iqn.2003-01....l. Feb 07 14:45:11 debian-sanboot logout-all.sh[971]: Logging out of session [sid: 4, target:...0] Feb 07 14:45:11 debian-sanboot logout-all.sh[971]: Logout of [sid: 4, target: iqn.2003-01....l. Feb 07 14:45:11 debian-sanboot systemd[1]: Stopped Login to default iSCSI targets. Hint: Some lines were ellipsized, use -l to show in full. It terminated all the connections, which it should not. root@debian-sanboot:~# systemctl stop open-iscsi root@debian-sanboot:~# iscsiadm -m session iscsiadm: No active sessions. > Now for the problem in your case: there is code in the open-iscsi > package in unstable that tries to make sure we don't close > sessions where the rootfs is on. This consists of two steps: > > 1) /lib/open-iscsi/umountiscsi.sh [#] will try to umount > filesystems on iSCSI. At the same time it detects sessions that > contain the root filesystem and writes them out into the file > /run/open-iscsi/shutdown-keep-sessions. > Looks like this invocation is problematic root@debian-sanboot:~# /lib/open-iscsi/umountiscsi.sh ls: cannot access /dev/sda?*: No such file or directory ls: cannot access /dev/sdb?*: No such file or directory ls: cannot access /dev/sdc?*: No such file or directory ls: cannot access /dev/sdd?*: No such file or directory > 2) /lib/open-iscsi/logout-all.sh [~] will detect if the file > /run/open-iscsi/shutdown-keep-sessions exists - and if it does > it will *NOT* run the default iscsiadm -m node --logoutall=all > but rather log out of all sessions that were not excluded. > > So what happens in your case is that umountiscsi.sh likely > doesn't properly detect your setup where your rootfs is on > iSCSI - and therefore doesn't exclude the relevant sessions > from logout. > > To test this, you should run the /lib/open-iscsi/umountiscsi.sh > script manually - that should *not* try to umount anything > (because it should detect that it's the rootfs) and write out > /run/open-iscsi/shutdown-keep-sessions. You should check that > that file contains all the sessions that your rootfs relies > on. > > Hope that gives you enough pointers to debug this. > From what you've explained, this is what I've gotten below. root@debian-sanboot:~# bash /lib/open-iscsi/umountiscsi.sh ^C root@debian-sanboot:~# cat /run/open-iscsi/shutdown-keep-sessions /sys/devices/platform/host2/session1 /sys/devices/platform/host3/session2 /sys/devices/platform/host4/session3 /sys/devices/platform/host5/session4 root@debian-sanboot:~# iscsiadm -m session tcp: [1] 172.16.20.40:3260,1 iqn.2003-01.org.linux-iscsi.debian.sanboot (non-flash) tcp: [2] 172.16.20.41:3260,1 iqn.2003-01.org.linux-iscsi.debian.sanboot (non-flash) tcp: [3] 172.16.20.42:3260,1 iqn.2003-01.org.linux-iscsi.debian.sanboot (non-flash) tcp: [4] 172.16.20.43:3260,1 iqn.2003-01.org.linux-iscsi.debian.sanboot (non-flash) But with all this setting, it hangs the shutdown, because all session are then terminated. > Regards, > Christian > > [*] systemd tracks processes, so it's going to be killed. There > is a possibility to avoid this, but this requires changes to > iscsid and will be rather complicated to get the integration > right. Not saying we shouldn't consider this, but just saying > that's complicated, and is something that's much lower on my > TODO list. > > [#] http://anonscm.debian.org/cgit/pkg-iscsi/open-iscsi.git/tree/debi > an/extra/umountiscsi.sh > > [~] http://anonscm.debian.org/cgit/pkg-iscsi/open-iscsi.git/tree/debi > an/extra/logout-all.sh > root@debian-sanboot:~# systemctl stop open-iscsi root@debian-sanboot:~# cat /run/open-iscsi/shutdown-keep-sessions /sys/devices/platform/host2/session1 /sys/devices/platform/host3/session2 /sys/devices/platform/host4/session3 /sys/devices/platform/host5/session4 root@debian-sanboot:~# iscsiadm -m session iscsiadm: No active sessions. From the workflow you've explained so far, the mention of certain sessions in /run/o-i/s-k-s should have been honored. -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System
signature.asc
Description: This is a digitally signed message part
