Package: open-iscsi
Version: 2.0.873+git0.3b4b4500-4
Severity: important
Tags: patch
Dear Maintainer,
tl;dr: systemd + open-iscsi = 90s hang at boot in some cases,
and umountiscsi.sh is not called on shutdown. Attached a
debdiff that fixes that without being too invasive.
Longer explanation: if you have the following configuration:
- Jessie
- systemd as init
- open-iscsi configured to automatically log in to some iSCSI target,
iSCSI disk /dev/sdb is then available
- /etc/fstab containing an entry like
/dev/sdb1 /data ext4 rw,_netdev 0 0
or (when using LVM)
/dev/vg_.../lv_... /data ext4 rw,_netdev 0 0
the system boot will hang for 90s because of systemd's default timeout
when devices are not available.
The reason behind this is that open-iscsi contains the following LSB
headers:
Required-Start: $network $remote_fs
Required-Stop: $network $remote_fs sendsigs
Here, $network maps to network-online.target in systemd, that's fine,
but $remote_fs maps to remote-fs.target in systemd, that is the problem.
This is because
a) systemd treats file systems that couldn't be mounted as hard
failures.
and
b) systemd's logic of mounting all remote filesystems is to mount
all filesystems in /etc/fstab that are marked _netdev (and not
makred noauto)
Therefore, systemd waits for the iSCSI device to appear for 90s before
timing out and proceeding with boot. Only then remote-fs.target is
reached and systemd starts the open-iscsi init script.
That in turn will then make the devices appear. The init script will
then call a "mount -a -O _netdev" and "swapon -a -e" in it's start()
routine, that will then cause the mount points to be activated.
So in the end, the boot is kind-of successful in the sense that
everything kind of works at the end of boot, with the following two caveats:
- there is this needless 90s delay (or whatever other delay the admin
has configured) in waiting on the iSCSI targets
- if I want to use systemd's features to order to order a specific
service after remote-fs.target to make sure that the remove file
systems I have are mounted, maybe because the service needs the
data on them, then this won't work consistently, because the
file systems will only be mounted after open-iscsi is started,
which will then be in parallel to any services I have ordered
after remote-fs.target, for example:
- exporting a subdirectory of an iSCSI filesystem via NFS; if
nfs-kernel-server gets started too early, this might fail
because the directory that is exported doesn't exist
If I modify the init script to remove $remote_fs from it's LSB headers,
then booting works as expected. However, this causes two problems:
1. I assume that $remote_fs is in there because you want to support
NFS-based sepearte /usr. Removing $remote_fs from LSB headers
would break such a configuration under sysvinit, since the
open-iscsi tools wouldn't be able to be called.
However, systemd in Debian currently doesn't really support a
separate /usr that's not mounted from initrd anyway.
2. Shutting down is racy.
Shutting down is racy because you then have the following constellation:
- systemd tracks services' states. And while bug #732793 does not occur
anymore because invoke-rc.d strips the .sh from umountiscsi.sh, the
call to umountiscsi.sh stop doesn't really do anything, because
systemd already thinks it is stopped, since it was never started.
- OTOH, systemd will tear down remote filesystems on its own. But
because open-iscsi is only ordered after network-online.target then,
tearing down the remote filesystesm will be done in parallel (!) to
stopping open-iscsi.
This has the unfortunate effect that it could be the case that the
umount call to the filesystem is made after open-iscsi has been
stopped. This will then cause the kernel to hang trying to umount
the filesystem.
I haven't been able to reproduce this race yet, i.e. I have gotten
lucky so far, in that umount was typically faster on my system than
stopping open-iscsi - BUT I am really not comfortable with having
such a flimsy race in place, especially since umount will sync
stuff to the filesystem and stopping open-iscsi too early could
easily cause severe data loss.
So far for my analysis. How do we proceed from here?
- it is quite clear that you probably don't want to change the sysvinit
logic now, especially so late in the Jessie freeze
- however, this bug w.r.t. systemd should definitely be fixed in my
eyes
Therefore, I suggest that you provide a unit file specifically for
systemd. In order to as minimally invasive as possible (especially this
late in the freeze), the unit file should ideally call the original init
script.
After Jessie one should consider redoing the entire logic for
systemd-based systems, there are a lot more features of sytstemd that
one can leverage to make things work better. But to fix this immediate
bug, the changes I mentioned are sufficient.
I have created a debdiff for a test package that changes the following:
- add systemd unit that just calls the init script but has adjusted
dependencies:
- no more After=remote-fs.target
- new Before=remote-fs-pre.target
- add dh-systemd as build-dep and use dh_systemd in debian/rules
- move #DEBHELPER# around in postinst, to make sure package upgrades
don't break the system (dh_systemd_enable code has to come before
the unit is first started, otherwise weird things occur)
- do the equivalent of umountiscsi.sh start so that systemd will
track that service as 'running' - then at shutdown the open-iscsi
init script will be able to call the stop action of that script
I have now tested this under systemd with Jessie, in two different
configurations of Jessie running systemd:
1. root on normal device, separate iSCSI devices mounted
2. root on iSCSI, boot via PXE
In both cases, iSCSI now seems to work as expected. There are a couple
of caveats though:
- as discussed before, non-initrd-mounted separate /usr on NFS
won't work together with this constellation
- unlikely to work well with systemd anyway, regardless of
iSCSI, and I don't think this is something that could be
fixed without a major redesign of the remote-fs*.target
logic across the board
- irrespective of systemd, while looking at it I noticed that
umountiscsi.sh's logic is incomplete, it doesn't try to umount
filesystems on LVM on top of iSCSI, unless they were marked with
_netdev (it only detects direct devices).
OTOH, this has been the case since at least Squeeze, so it can't
be that critical.
- the current design of using umountiscsi.sh doesn't integrate well
with systemd's dependency logic. I don't think this is a huge issue,
as far as I can see, stuff works as well under systemd with my patch
as under sysvinit (except for the /usr-NFS thing), but I do think
that you could make the whole thing a lot more robust if this is
redesigned a bit - but I don't think that is something that should
go to Jessie.
-- System Information:
Debian Release: 8.0
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 3.16.0-4-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Init: systemd (via /run/systemd/system)
Versions of packages open-iscsi depends on:
ii libc6 2.19-13
ii udev 215-8.0+~tfptkm1
open-iscsi recommends no packages.
open-iscsi suggests no packages.
-- no debconf information
Btw. I selected severity 'important' because I don't think this bug is
'grave', but I do think that it could be categorized as 'serious', since
in my eyes it is unwritten policy that packages should properly support
the default init system unless there's a really good reason against it.
Unfortunately for me, current policy doesn't mention multiple init
systems at all, therefore the severity 'important', because I can't
point to a specific part of the text. Nevertheless, I think this bug
would qualify as RC.
Regards,
Christian
diffstat for open-iscsi-2.0.873+git0.3b4b4500 open-iscsi-2.0.873+git0.3b4b4500
changelog | 14 ++++++++++++++
control | 2 +-
open-iscsi.init | 10 ++++++++++
open-iscsi.postinst | 4 ++--
open-iscsi.service | 22 ++++++++++++++++++++++
rules | 2 ++
6 files changed, 51 insertions(+), 3 deletions(-)
diff -Nru open-iscsi-2.0.873+git0.3b4b4500/debian/changelog
open-iscsi-2.0.873+git0.3b4b4500/debian/changelog
--- open-iscsi-2.0.873+git0.3b4b4500/debian/changelog 2014-09-01
11:03:23.000000000 +0200
+++ open-iscsi-2.0.873+git0.3b4b4500/debian/changelog 2015-01-19
19:45:36.000000000 +0100
@@ -1,3 +1,17 @@
+open-iscsi (2.0.873+git0.3b4b4500-4.0+systemd1) UNRELEASED; urgency=medium
+
+ * Test package
+ * Create systemd unit, make it order before remote-fs-pre.target to
+ fix hang at boot. Unit currently only starts init script.
+ * Manually start umountiscsi.sh in open-iscsi init script to make
+ the stop action on shutdown not be a noop. (systemd tracks service
+ state)
+ * Add dh-systemd to build-deps.
+ * Reorder #DEBHELPER# in postinst to not break upgrades (dh-systemd's
+ code has to be there before invoke-rc.d is called).
+
+ -- Christian Seiler <christ...@iwakd.de> Mon, 19 Jan 2015 19:45:05 +0100
+
open-iscsi (2.0.873+git0.3b4b4500-4) unstable; urgency=medium
* [41c7eca] Introduce new architectures based on current build
diff -Nru open-iscsi-2.0.873+git0.3b4b4500/debian/control
open-iscsi-2.0.873+git0.3b4b4500/debian/control
--- open-iscsi-2.0.873+git0.3b4b4500/debian/control 2014-09-01
11:02:01.000000000 +0200
+++ open-iscsi-2.0.873+git0.3b4b4500/debian/control 2015-01-19
18:59:21.000000000 +0100
@@ -3,7 +3,7 @@
Priority: optional
Maintainer: Debian iSCSI Maintainers
<pkg-iscsi-maintain...@lists.alioth.debian.org>
Uploaders: Andrew Moise <ch...@demiurgestudios.com>, Philipp Hug
<deb...@hug.cx>, Guido Günther <a...@sigxcpu.org>, Ritesh Raj Sarraf
<r...@debian.org>
-Build-Depends: debhelper (>= 7.0.0), bzip2, bison, flex, autotools-dev,
dh-autoreconf, dpkg-dev (>= 1.16.1~)
+Build-Depends: debhelper (>= 7.0.0), bzip2, bison, flex, autotools-dev,
dh-autoreconf, dpkg-dev (>= 1.16.1~), dh-systemd
Standards-Version: 3.9.2
Vcs-Git: git://anonscm.debian.org/pkg-iscsi/open-iscsi.git
Vcs-Browser: http://anonscm.debian.org/gitweb/?p=pkg-iscsi/open-iscsi.git
diff -Nru open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.init
open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.init
--- open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.init 2014-08-20
15:53:55.000000000 +0200
+++ open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.init 2015-01-19
19:45:04.000000000 +0100
@@ -107,6 +107,16 @@
udevadm settle || true;
+ # If we are under systemd, make sure we start umountiscsi.sh.
+ # This is a no-op, but systemd tracks each unit's status, so
+ # we unless we start it here, the invoke-rc.d during stop()
+ # doesn't do anything.
+ if [ -d /run/systemd/system ] ; then
+ # we don't want to deadlock here, so ignore deps,
+ # the start of the service is a no-op anyway
+ systemctl --job-mode=ignore-dependencies start
umountiscsi.service
+ fi
+
# Handle iSCSI LVM devices
if [ ! -x "/sbin/vgchange" -a -n "$LVMGROUPS" ]; then
diff -Nru open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.postinst
open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.postinst
--- open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.postinst 2014-08-20
15:53:55.000000000 +0200
+++ open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.postinst 2015-01-19
19:04:03.000000000 +0100
@@ -11,6 +11,8 @@
fi
}
+#DEBHELPER#
+
case "$1" in
configure)
# Move old configuration from /etc/ into /etc/iscsi/
@@ -43,6 +45,4 @@
;;
esac
-#DEBHELPER#
-
exit 0
diff -Nru open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.service
open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.service
--- open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.service 1970-01-01
01:00:00.000000000 +0100
+++ open-iscsi-2.0.873+git0.3b4b4500/debian/open-iscsi.service 2015-01-19
19:14:53.000000000 +0100
@@ -0,0 +1,22 @@
+[Unit]
+Description=iSCSI initiator
+DefaultDependencies=no
+Before=sysinit.target shutdown.target remote-fs-pre.target
+After=network-online.target
+Wants=network-online.target
+Conflicts=shutdown.target
+
+[Service]
+Type=forking
+Restart=no
+TimeoutSec=0
+IgnoreSIGPIPE=no
+KillMode=process
+GuessMainPID=no
+RemainAfterExit=yes
+SysVStartPriority=20
+ExecStart=/etc/init.d/open-iscsi start
+ExecStop=/etc/init.d/open-iscsi stop
+
+[Install]
+WantedBy=multi-user.target
diff -Nru open-iscsi-2.0.873+git0.3b4b4500/debian/rules
open-iscsi-2.0.873+git0.3b4b4500/debian/rules
--- open-iscsi-2.0.873+git0.3b4b4500/debian/rules 2014-08-20
15:53:55.000000000 +0200
+++ open-iscsi-2.0.873+git0.3b4b4500/debian/rules 2015-01-19
19:18:24.000000000 +0100
@@ -127,8 +127,10 @@
dh_installchangelogs
dh_installdocs
dh_installexamples
+ dh_systemd_enable
dh_installinit -u 'start 45 S . stop 81 0 1 6 .' --no-start
dh_installinit -u 'stop 80 0 1 6 .' --no-start --name=umountiscsi.sh
+ dh_systemd_start --no-start
dh_installman
dh_link
dh_strip
