Package: multipath-tools
Version: 0.9.4-3+deb12u1
Severity: important
Dear Maintainer,
I had to do some minor changes to multipath-tools to fresh install a new
system on SAN disks only.
These I want to report. The changes are:
1. Add conflict= lines to multipathd.socket file for systemd
2. ln -s /lib/multipath /lib64
Installation image was debian-12.6.0-amd64-netinst.iso
Host bus adapter is Fibre Channel: Emulex Corporation LPe31000/LPe32000
SAN Storage is a Dell Compellant with 5 LUNs, used as:
- /boot + EFI Partition
- /
- /tmp
- /var
- swap
1. I decided not to use the default initramfs-tools as it seems to miss
code that checks /etc/fstab and waits for all required FC LUN's to
appear (which usually takes up to 7-9s at my site), so system start
always fails.
Instead I installed dracut as initrd. To my observations it waits
reliable for the required LUN's to appear but afterwards hangs forever
in initrd-cleanup.
After some investigation it seems that dracut stops multipathd after LUN
detection completes, but it immediately gets restarted again by
multipathd.socket so blocking the state transition of initrd-cleanup.
After adding the same conflict= lines from multipathd.service to
multipathd.socket and doing a dracut -f this does not happen any more.
--- lib/systemd/system/multipathd.socket.orig 2024-08-08
15:29:15.193540154 +0200
+++ lib/systemd/system/multipathd.socket 2024-08-08
15:29:21.417542212 +0200
@@ -1,6 +1,8 @@
[Unit]
Description=multipathd control socket
DefaultDependencies=no
+Conflicts=shutdown.target
+Conflicts=initrd-cleanup.service
ConditionKernelCommandLine=!nompath
ConditionKernelCommandLine=!multipath=off
ConditionVirtualization=!container
Keeping Logfiles from inside the hanging initrd was very difficult for
me, I tried to put
rd.debug rd.info rd.break
to kernel cmdline and somehow managed to save some log information to an
extra partition. If you need more logs please let me know.
Some output from the logfile I managed to keep:
multipathd get's stopped:
[ 9.029421] localhost systemd[1]: multipathd.service: Job 156
multipathd.service/stop finished, result=done
multipathd.socket changes to listening:
[ 9.029454] localhost systemd[1]: multipathd.socket: Changed running
-> listening
And systemd gets traffic on the socket and starts the service again:
[ 9.029535] localhost systemd[1]: multipathd.socket: Incoming traffic
[ 9.029549] localhost systemd[1]: multipathd.service: Trying to
enqueue job multipathd.service/start/replace
[ 9.029590] localhost systemd[1]: multipathd.service: Installed new
job multipathd.service/start as 157
[ 9.029603] localhost systemd[1]: multipathd.service: Enqueued job
multipathd.service/start as 157
[ 9.029616] localhost systemd[1]: multipathd.socket: Changed
listening -> running
This is the output from "grep multipathd logfile". Let me know if you
need the whole file.
[ 2.076017] localhost systemd[1]: unit_file_build_name_map: normal
unit file: /lib/systemd/system/multipathd.socket
[ 2.076025] localhost systemd[1]: unit_file_build_name_map: normal
unit file: /lib/systemd/system/multipathd.service
[ 2.101366] localhost systemd[1]: multipathd.service: Installed new
job multipathd.service/start as 27
[ 2.361731] localhost (modp[229]: multipathd.service: Executing:
/sbin/modprobe dm-multipath
[ 2.574658] localhost multipathd[287]: multipathd v0.9.4: start up
[ 2.574658] localhost multipathd[287]: reconfigure: setting up paths
and maps
[ 3.763827] localhost (ltip[287]: multipathd.service: Executing:
/sbin/multipathd -d -s
[ 3.764262] localhost systemd[1]: multipathd.service: Got
notification message from PID 287 (STATUS=startup)
[ 3.764275] localhost systemd[1]: multipathd.service: Got
notification message from PID 287 (STATUS=configure)
[ 3.765260] localhost systemd[1]: multipathd.service: Got
notification message from PID 287 (STATUS=up)
[ 3.765272] localhost systemd[1]: multipathd.service: Got
notification message from PID 287 (READY=1)
[ 3.765285] localhost systemd[1]: multipathd.service: Changed start
-> running
[ 3.765298] localhost systemd[1]: multipathd.service: Job 27
multipathd.service/start finished, result=done
[ 3.765312] localhost systemd[1]: Started multipathd.service -
Device-Mapper Multipath Device Controller.
[ 3.765448] localhost systemd[1]: multipathd.socket: Changed
listening -> running
[ 6.725648] localhost multipathd[287]: mail6_tmp: addmap [0 20971520
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 1 1 8:48 1]
[ 6.823606] localhost multipathd[287]: sdd [8:48]: path added to
devmap mail6_tmp
[ 6.823606] localhost multipathd[287]: mail6_var: addmap [0 155189248
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 1 1 8:64 1]
[ 6.856524] localhost multipathd[287]: sde [8:64]: path added to
devmap mail6_var
[ 6.861086] localhost multipathd[287]: mail6_boot: addmap [0 2097152
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 1 1 8:0 1]
[ 6.932086] localhost multipathd[287]: sda [8:0]: path added to
devmap mail6_boot
[ 6.932086] localhost multipathd[287]: mail6_swap: addmap [0 4194304
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 1 1 8:32 1]
[ 6.968139] localhost multipathd[287]: sdc [8:32]: path added to
devmap mail6_swap
[ 7.466211] localhost multipathd[287]: mail6_boot: reload [0 2097152
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:0 1 8:80 1]
[ 7.508901] localhost multipathd[287]: sdf [8:80]: path added to
devmap mail6_boot
[ 7.516745] localhost multipathd[287]: mail6_tmp: reload [0 20971520
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:48 1 8:128 1]
[ 7.569157] localhost multipathd[287]: sdi [8:128]: path added to
devmap mail6_tmp
[ 7.577696] localhost multipathd[287]: mail6_swap: reload [0 4194304
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:32 1 8:112 1]
[ 7.634478] localhost multipathd[287]: sdh [8:112]: path added to
devmap mail6_swap
[ 7.663410] localhost multipathd[287]: mail6_root: addmap [0 33554432
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:16 1 8:96 1]
[ 7.706075] localhost multipathd[287]: sdg [8:96]: path added to
devmap mail6_root
[ 7.728204] localhost multipathd[287]: mail6_var: reload [0 155189248
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:64 1 8:144 1]
[ 7.752941] localhost multipathd[287]: sdj [8:144]: path added to
devmap mail6_var
[ 8.773304] localhost systemd[1]: multipathd.service: Installed new
job multipathd.service/stop as 156
[ 8.773684] localhost systemd[1]: Sent message type=signal
sender=org.freedesktop.systemd1 destination=n/a
path=/org/freedesktop/systemd1/unit/multipathd_2eservice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=5 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 8.773782] localhost systemd[1]: Sent message type=signal
sender=org.freedesktop.systemd1 destination=n/a
path=/org/freedesktop/systemd1/unit/multipathd_2eservice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=6 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 8.774157] localhost systemd[1]: multipathd.service: Thawing unit.
[ 8.774210] localhost systemd[1]: Sent message type=signal
sender=org.freedesktop.systemd1 destination=n/a
path=/org/freedesktop/systemd1/unit/multipathd_2eservice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=9 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 8.774257] localhost multipathd[287]: multipathd: shut down
[ 8.775151] localhost systemd[1]: Sent message type=signal
sender=org.freedesktop.systemd1 destination=n/a
path=/org/freedesktop/systemd1/unit/multipathd_2eservice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=10 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 8.775360] localhost systemd[1]: multipathd.service: Changed running
-> stop-sigterm
[ 8.775452] localhost systemd[1]: Stopping multipathd.service -
Device-Mapper Multipath Device Controller...
[ 8.777035] localhost systemd[1]: Sent message type=signal
sender=org.freedesktop.systemd1 destination=n/a
path=/org/freedesktop/systemd1/unit/multipathd_2eservice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=11 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 8.777214] localhost systemd[1]: Sent message type=signal
sender=org.freedesktop.systemd1 destination=n/a
path=/org/freedesktop/systemd1/unit/multipathd_2eservice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=12 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 8.777322] localhost systemd[1]: multipathd.service: Got
notification message from PID 287 (STATUS=shutdown)
[ 8.777376] localhost systemd[1]: Sent message type=signal
sender=org.freedesktop.systemd1 destination=n/a
path=/org/freedesktop/systemd1/unit/multipathd_2eservice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=14 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 8.777421] localhost systemd[1]: Sent message type=signal
sender=org.freedesktop.systemd1 destination=n/a
path=/org/freedesktop/systemd1/unit/multipathd_2eservice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=15 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 8.777481] localhost systemd[1]: multipathd.service: Got
notification message from PID 287 (STOPPING=1)
[ 8.777528] localhost systemd[1]: Sent message type=signal
sender=org.freedesktop.systemd1 destination=n/a
path=/org/freedesktop/systemd1/unit/multipathd_2eservice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=16 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 8.777576] localhost systemd[1]: Sent message type=signal
sender=org.freedesktop.systemd1 destination=n/a
path=/org/freedesktop/systemd1/unit/multipathd_2eservice
interface=org.freedesktop.DBus.Properties member=PropertiesChanged
cookie=17 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 8.777619] localhost systemd[1]: multipathd.service: Got
notification message from PID 287 (ERRNO=0)
[ 8.980593] localhost systemd[1]: Received SIGCHLD from PID 287
(multipathd).
[ 8.988095] localhost systemd[1]: Child 287 (multipathd) died
(code=exited, status=0/SUCCESS)
[ 8.988190] localhost dracut-pre-pivot[950]: + [ -e
//lib/dracut/hooks/cleanup/80-multipathd-needshutdown.sh ]
[ 8.988190] localhost dracut-pre-pivot[950]: + .
//lib/dracut/hooks/cleanup/80-multipathd-needshutdown.sh
[ 9.027334] localhost systemd[1]: multipathd.service: Child 287
belongs to multipathd.service.
[ 9.027367] localhost multipathd[962]: multipathd v0.9.4: start up
[ 9.027367] localhost multipathd[962]: reconfigure: setting up paths
and maps
[ 9.027367] localhost multipathd[962]: mail6_boot: reload [0 2097152
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:0 1 8:80 1]
[ 9.029347] localhost systemd[1]: multipathd.service: Main process
exited, code=exited, status=0/SUCCESS (success)
[ 9.029371] localhost systemd[1]: multipathd.service: Deactivated
successfully.
[ 9.029391] localhost systemd[1]: multipathd.service: Service restart
not allowed.
[ 9.029406] localhost systemd[1]: multipathd.service: Changed
stop-sigterm -> dead
[ 9.029421] localhost systemd[1]: multipathd.service: Job 156
multipathd.service/stop finished, result=done
[ 9.029436] localhost systemd[1]: Stopped multipathd.service -
Device-Mapper Multipath Device Controller.
[ 9.029454] localhost systemd[1]: multipathd.socket: Changed running
-> listening
[ 9.029470] localhost systemd[1]: multipathd.service: Consumed 273ms
CPU time.
[ 9.029488] localhost systemd[1]: Cannot stat
/run/credentials/multipathd.service: No such file or directory
[ 9.029503] localhost systemd[1]: multipathd.service: Control group
is empty.
[ 9.029535] localhost systemd[1]: multipathd.socket: Incoming traffic
[ 9.029549] localhost systemd[1]: multipathd.service: Trying to
enqueue job multipathd.service/start/replace
[ 9.029590] localhost systemd[1]: multipathd.service: Installed new
job multipathd.service/start as 157
[ 9.029603] localhost systemd[1]: multipathd.service: Enqueued job
multipathd.service/start as 157
[ 9.029616] localhost systemd[1]: multipathd.socket: Changed
listening -> running
[ 9.029644] localhost systemd[1]: multipathd.service:
ConditionVirtualization=!container succeeded.
[ 9.029658] localhost systemd[1]: multipathd.service:
ConditionKernelCommandLine=!multipath=off succeeded.
[ 9.029671] localhost systemd[1]: multipathd.service:
ConditionKernelCommandLine=!rd_NO_MULTIPATH succeeded.
[ 9.029684] localhost systemd[1]: multipathd.service:
ConditionKernelCommandLine=!rd.multipath=0 succeeded.
[ 9.029696] localhost systemd[1]: multipathd.service:
ConditionKernelCommandLine=!nompath succeeded.
[ 9.029709] localhost systemd[1]: multipathd.service: Will spawn
child (service_enter_start_pre): /sbin/modprobe
[ 9.029726] localhost systemd[1]: multipathd.service: About to
execute /sbin/modprobe dm-multipath
[ 9.029743] localhost systemd[1]: multipathd.service: Forked
/sbin/modprobe as 961
[ 9.029757] localhost systemd[1]: multipathd.service: Changed dead ->
start-pre
[ 9.029769] localhost systemd[1]: Starting multipathd.service -
Device-Mapper Multipath Device Controller...
[ 9.029785] localhost (modp[961]: multipathd.service: Executing:
/sbin/modprobe dm-multipath
[ 9.029846] localhost systemd[1]: multipathd.service: Child 961
belongs to multipathd.service.
[ 9.029862] localhost systemd[1]: multipathd.service: Control process
exited, code=exited, status=0/SUCCESS (success)
[ 9.029878] localhost systemd[1]: multipathd.service: Got final
SIGCHLD for state start-pre.
[ 9.029893] localhost systemd[1]: multipathd.service: Will spawn
child (service_enter_start): /sbin/multipathd
[ 9.029906] localhost systemd[1]: multipathd.service: Passing 1 fds
to service
[ 9.029920] localhost systemd[1]: multipathd.service: About to
execute /sbin/multipathd -d -s
[ 9.029933] localhost systemd[1]: multipathd.service: Forked
/sbin/multipathd as 962
[ 9.029946] localhost systemd[1]: multipathd.service: Changed
start-pre -> start
[ 9.029958] localhost (ltip[962]: multipathd.service: Executing:
/sbin/multipathd -d -s
[ 9.029978] localhost systemd[1]: multipathd.service: Got
notification message from PID 962 (STATUS=startup)
[ 9.029992] localhost systemd[1]: multipathd.service: Got
notification message from PID 962 (STATUS=configure)
[ 9.037776] localhost multipathd[962]: mail6_root: reload [0 33554432
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:16 1 8:96 1]
[ 9.093797] localhost multipathd[962]: mail6_swap: reload [0 4194304
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:32 1 8:112 1]
[ 9.146576] localhost multipathd[962]: mail6_tmp: reload [0 20971520
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:48 1 8:128 1]
[ 9.213730] localhost multipathd[962]: mail6_var: reload [0 155189248
multipath 1 queue_if_no_path 1 alua 1 1 service-time 0 2 1 8:64 1 8:144 1]
[ 9.253482] localhost systemd[1]: multipathd.service: Got
notification message from PID 962 (STATUS=up)
[ 9.253561] localhost systemd[1]: multipathd.service: Got
notification message from PID 962 (READY=1)
[ 9.253622] localhost systemd[1]: multipathd.service: Changed start
-> running
[ 9.253683] localhost systemd[1]: multipathd.service: Job 157
multipathd.service/start finished, result=done
[ 9.253799] localhost systemd[1]: Started multipathd.service -
Device-Mapper Multipath Device Controller.
2. dracut does not search for the shared libraries for multipathd at
/lib/multipath
but at
/lib64/multipath
so the initrd does not contain the required shared libraries.
I'm unsure if it's a bug of dracut or multipath-tools.
Setting a softlink /lib64/multipath -> /lib/multipath/ solves it.
Best regards,
Th. Schlichting
