Package: open-iscsi Version: 2.1.11-1+deb13u1 Severity: normal
Dear Maintainer, We have several diskless servers with iSCSI root arranged in the stack:btrfs : lvm : gpt : multipath : iscsi. The btrfs filesystem contains subvolumes mounted at / /boot and /var.
The servers host virtual machines with storage arranged as follows:lvm : multipath : iscsi. The connections to the iscsi targets use vlan-bonding interfaces, where the vlan id for the iSCSI root is different to the one used for virtual disks storage.
This scheme has proved to be reliable over several years and the stability and robustness of open-iscsi is essential for its successful performance. Thanks for your effort in maintaining the quality of the open-iscsi package.
However, at reboot or shutdown the servers were consistently freezing trying to synchronize iscsi devices. After some tests we have recently solved the issue with a modified umountiscsi.sh script. Patch series is attached.
In the tests it was found that:1. The umountiscsi.sh script tried to disassemble the multipath device that supports the iSCSI root and failed. It was unable to identify that device as support for the root filesystem, exclude it from the disassembly process and add the relevant sessions to the shutdown-keep-sessions file.
2. After umountiscsi.sh exited with an error status the logout-all.sh script is not executed, leaving all iscsi connections open. The open-iscsi.service stop operation exited with failed status.
3. The vlan-bonding interface that supports the virtual machines iscsi storage is deactivated as part of the server shutdown sequence. At this point this interface should have been free because the relevant iscsi sessions were supossed to be closed by the open-iscsi stop operation. The vlan-bonding interface that supports the iscsi root is configured to be kept active at shutdown.
4. The server's kernel tries to synchronize iscsi devices near the end of the shutdown sequence and only succeeds with those involved with the iscsi root. For the rest of iscsi devices connection errors are reported and the shutdown process freezes.
This long standing issue was solved with the following umountiscsi.sh changes:
1. Create directory for signaling file 2. Fix detection of LVM VGs with iSCSI storage 3. Add the path to /var/log/journal to EXCLUDED_MOUNTS 4. Use lsblk to find more iscsi mount points 5. Use lsblk to find the major:minor number of a device 6. Remove unused list iscsi_mount_point_idsPatches 2, 3 and 4 implement the core of the changes. Patch 5 contains a proposed improvement and patches 1 and 6 address what seem to be minor details.
We hope this patch series can be useful. Best regards, -- System Information: Debian Release: 13.1 APT prefers stable-updatesAPT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64) Kernel: Linux 6.12.43+deb13-amd64 (SMP w/64 CPU threads; PREEMPT)Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en
Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages open-iscsi depends on: ii debconf [debconf-2.0] 1.5.91 ii init-system-helpers 1.69~deb13u1 ii libc6 2.41-12 ii libisns0t64 0.101-1+b1 ii libkmod2 34.2-2 ii libmount1 2.41-5 ii libopeniscsiusr 2.1.11-1+deb13u1 ii libssl3t64 3.5.1-1 ii libsystemd0 257.8-1~deb13u2 ii udev 257.8-1~deb13u2 Versions of packages open-iscsi recommends: ii busybox 1:1.37.0-6+b3 open-iscsi suggests no packages. -- debconf information: open-iscsi/upgrade_even_with_failed_sessions: open-iscsi/downgrade_and_break_system: open-iscsi/remove_even_with_active_sessions: open-iscsi/upgrade_recovery_error:
From f740cec5f0b1d93993c04d69937b5e723a07f45e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dhionel=20D=C3=ADaz?= <[email protected]> Date: Wed, 17 Sep 2025 11:29:18 -0400 Subject: [PATCH 1/6] umountiscsi.sh: create directory for signaling file If there are no excluded sessions the directory /run/open-iscsi may not be present at the point when a signaling file is going to be created. --- debian/extra/umountiscsi.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/debian/extra/umountiscsi.sh b/debian/extra/umountiscsi.sh index 9c6dfbc..752857a 100755 --- a/debian/extra/umountiscsi.sh +++ b/debian/extra/umountiscsi.sh @@ -666,6 +666,7 @@ done # Create signaling file (might be useful) if [ $any_umount_failed -eq 1 ] ; then + mkdir -p -m 0700 /run/open-iscsi touch /run/open-iscsi/some_umount_failed else rm -f /run/open-iscsi/some_umount_failed -- 2.51.0
From 2342b6df2b31f925e137ffe629f3dd7523a13108 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dhionel=20D=C3=ADaz?= <[email protected]> Date: Wed, 17 Sep 2025 12:32:13 -0400 Subject: [PATCH 2/6] umountiscsi.sh: fix detection of LVM VGs with iSCSI storage The pvs command seems to be expecting Physical Volumes to be specified by absolute device paths. When relative paths are provided the command fails and the LVM volume groups with iSCSI storage are not detected. --- debian/extra/umountiscsi.sh | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/debian/extra/umountiscsi.sh b/debian/extra/umountiscsi.sh index 752857a..eca2ec6 100755 --- a/debian/extra/umountiscsi.sh +++ b/debian/extra/umountiscsi.sh @@ -192,11 +192,8 @@ get_lvm_vgs() { # handle the case where we didn't get passed any PVs # at all [ $# -gt 0 ] || return 0 - # subshell for pwd change - ( - cd /dev - $PVS --noheadings -o vg_name "$@" 2>/dev/null - ) + + $PVS --noheadings -o vg_name $(printf '/dev/%s ' "$@") 2>/dev/null } enumerate_luks() { -- 2.51.0
From d1beecf080a085da63055d41f1963e17af9fcf40 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dhionel=20D=C3=ADaz?= <[email protected]> Date: Fri, 19 Sep 2025 13:34:58 -0400 Subject: [PATCH 3/6] umountiscsi.sh: add the path to /var/log/journal to EXCLUDED_MOUNTS In a systemd system the the existence of /var/log/journal directory indicates that a running journald daemon may be using it when umountiscsi.sh is called, therefore /var /var/log and /var/log/journal are excluded to avoid an umount error when a iscsi device is involved. When umountiscsi.sh exits with an error status logout-all.sh is not called and that may generate large delays or a freeze when the server tries to synchronize the devices at shutdown or reboot. --- debian/extra/umountiscsi.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/debian/extra/umountiscsi.sh b/debian/extra/umountiscsi.sh index eca2ec6..8a0b641 100755 --- a/debian/extra/umountiscsi.sh +++ b/debian/extra/umountiscsi.sh @@ -72,10 +72,14 @@ DRY_RUN=0 # We need to make sure that we don't try to umount the root device # and for systemd systems, also /usr (which is pre-mounted in initrd -# there). +# there) and the path to /var/log/journal when this directory +# exists. EXCLUDE_MOUNTS="/" if [ -d /run/systemd/system ] ; then EXCLUDE_MOUNTS="$EXCLUDE_MOUNTS /usr" + if [ -d /var/log/journal ] ; then + EXCLUDE_MOUNTS="$EXCLUDE_MOUNTS /var /var/log /var/log/journal" + fi fi EXCLUDE_MOUNTS="${EXCLUDE_MOUNTS}${EXCLUDE_MOUNTS_AT_SHUTDOWN+ $EXCLUDE_MOUNTS_AT_SHUTDOWN}" unset _EXCLUDE_MOUNTS -- 2.51.0
From 28df1b8f795645f21759b8583798519e47d1d183 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dhionel=20D=C3=ADaz?= <[email protected]> Date: Fri, 19 Sep 2025 14:35:32 -0400 Subject: [PATCH 4/6] umountiscsi.sh: use lsblk to find more iscsi mount points A mounted btrfs subvolume is listed with zero as device major number in /proc/self/mountinfo, preventing its identification as part of the iscsi_potential_mount_sources_majmin list. When the mountpoint is one of the EXCLUDED_MOUNTS that would also leave an incomplete list of sessions to exclude. The issue is solved using lsblk to find more iscsi mounts to deactivate and more sessions to exclude. The resulting iscsi_mount_points list is sorted in reverse order with duplicates removed. The situation where multiple mountpoints are using the same device is supported. The hexadecimal escape sequences that may use lsblk are changed to the octal equivalents, in order to keep congruence with /proc/self/mountinfo and compatibility with the dash shell. As the dash shell only supports octal escape sequences, the hex to octal conversion is performed by an awk script that uses the support for hexadecimal number strings provided in the POSIX compliant mode. The code uses lsblk, mawk, sed and sort, commands that should be available in this context as they are provided by required packages. --- debian/extra/umountiscsi.sh | 42 +++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/debian/extra/umountiscsi.sh b/debian/extra/umountiscsi.sh index 8a0b641..1bac84b 100755 --- a/debian/extra/umountiscsi.sh +++ b/debian/extra/umountiscsi.sh @@ -56,6 +56,8 @@ VGS=/sbin/vgs VGCHANGE=/sbin/vgchange CRYPTSETUP=/sbin/cryptsetup DMSETUP=/sbin/dmsetup +AWK="/usr/bin/mawk -Wposix" +LSBLK=/usr/bin/lsblk if [ -x $PVS ] && [ -x $LVS ] && [ -x $VGCHANGE ] ; then HAVE_LVM=1 @@ -180,6 +182,20 @@ hash_add_to_set() { add_to_set "${_hash_add_to_set_var}" "$@" } +hex_escape_to_octal_escape () { + _str=$( + printf '%s' "${2}" | $AWK '{ + i=$0; o=""; + while(r=match(i,/\\x([0-9a-fA-F]{2})/)){ + o=o substr(i,1,r) sprintf("%03o", "0" substr(i,r+1,3)); + i=substr(i,r+4); + } + print o i + }' + ) + eval $1=\${_str} +} + device_majmin() { eval $1=\"\" _majmin_dec=$(LC_ALL=C ls -lnd /dev/"$2" | while read _perms _links _uid _gid _majcomma _min _rest ; do @@ -393,6 +409,32 @@ enumerate_iscsi_devices() { iscsi_mount_point_ids="$_mpid $iscsi_mount_points" fi done < /proc/self/mountinfo + + # Find sessions to exclude using lsblk. Add iscsi mounts to iscsi_mount_points list. + # Handle the case where multiple mountpoints are reported for one device, separated + # by vertical whitespace. + while read -r _majmin _mnts; do + test -z "${_mnts}" && continue + if in_set iscsi_potential_mount_sources_majmin "${_majmin}" ; then + hex_escape_to_octal_escape _mnts "${_mnts}" + for _mntp in $(printf '%s' "${_mnts}" | sed 's/\\01[2-5]/ /g'); do + if in_set EXCLUDE_MOUNTS "${_mntp}" ; then + hash_get _dev_sessions ISCSI_NUMDEVICE_SESSIONS "${_majmin}" + add_to_set ISCSI_EXCLUDED_SESSIONS $_dev_sessions + else + # Push mountpoint to the top of the list + # to reverse the order provided by lsblk + iscsi_mount_points="${_mntp} $iscsi_mount_points" + fi + done + fi + done <<- EOF + $($LSBLK -r --noheadings -o MAJ:MIN,MOUNTPOINTS -s -x MOUNTPOINTS 2>/dev/null) + EOF + # Sort iscsi_mount_points list in reverse order and remove duplicates + iscsi_mount_points=$( + printf '%s ' $(printf '%s\n' ${iscsi_mount_points} | sort -r -u) | $AWK '{$1=$1}1' + ) } try_umount() { -- 2.51.0
From e03ca0746205a353040d8cb808ebeaa6979f714c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dhionel=20D=C3=ADaz?= <[email protected]> Date: Fri, 19 Sep 2025 15:56:20 -0400 Subject: [PATCH 5/6] umountiscsi.sh: use lsblk to find the major:minor number of a device The major:minor number of a device can be obtained directly using lsblk. This command should be available in this context because it's provided by a required package. --- debian/extra/umountiscsi.sh | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/debian/extra/umountiscsi.sh b/debian/extra/umountiscsi.sh index 1bac84b..85efab8 100755 --- a/debian/extra/umountiscsi.sh +++ b/debian/extra/umountiscsi.sh @@ -198,12 +198,7 @@ hex_escape_to_octal_escape () { device_majmin() { eval $1=\"\" - _majmin_dec=$(LC_ALL=C ls -lnd /dev/"$2" | while read _perms _links _uid _gid _majcomma _min _rest ; do - if [ x"${_majcomma%,}"x != x"${_majcomma}"x ] ; then - printf '%s' ${_majcomma%,}:${_min} - fi - break - done) + _majmin_dec=$($LSBLK --raw --noheadings --nodeps --output MAJ:MIN /dev/"$2" 2>/dev/null) [ -n "${_majmin_dec}" ] || return eval $1=\${_majmin_dec} } -- 2.51.0
From 9b738374390c5a4d7a572826056c28da725625fc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dhionel=20D=C3=ADaz?= <[email protected]> Date: Fri, 19 Sep 2025 16:07:22 -0400 Subject: [PATCH 6/6] umountiscsi.sh: remove unused list iscsi_mount_point_ids --- debian/extra/umountiscsi.sh | 2 -- 1 file changed, 2 deletions(-) diff --git a/debian/extra/umountiscsi.sh b/debian/extra/umountiscsi.sh index 85efab8..ef13192 100755 --- a/debian/extra/umountiscsi.sh +++ b/debian/extra/umountiscsi.sh @@ -387,7 +387,6 @@ enumerate_iscsi_devices() { # Enumerate mount points iscsi_mount_points="" - iscsi_mount_point_ids="" while read _mpid _mppid _mpdev _mpdevpath _mppath _mpopts _other ; do if in_set iscsi_potential_mount_sources_majmin "$_mpdev" ; then if in_set EXCLUDE_MOUNTS "${_mppath}" ; then @@ -401,7 +400,6 @@ enumerate_iscsi_devices() { # reflect the stacking order, so this is not # fool-proof, but it's better than nothing iscsi_mount_points="$_mppath $iscsi_mount_points" - iscsi_mount_point_ids="$_mpid $iscsi_mount_points" fi done < /proc/self/mountinfo -- 2.51.0
OpenPGP_signature.asc
Description: OpenPGP digital signature
