This bug was fixed in the package systemd - 249.11-0ubuntu3.10
---------------
systemd (249.11-0ubuntu3.10) jammy; urgency=medium
[ Nick Rosbrook ]
* debian/tests/tests-in-lxd: use --reuse flag in lxc publish (LP: #2023229)
File: debian/tests/tests-in-lxd
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=85b2ceddff1a6cc1ddbca8a1b7e5381d146e6313
* pstore: only try to load efi_pstore module (LP: #2023462)
File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7600bbfb1e8a399e5aeb1010a20deda3e5a06c89
* shutdown: get only active md arrays. (LP: #2025563)
File: debian/patches/lp2025563-shutdown-get-only-active-md-arrays.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=416a9245c8f0efbedcc4395cada23cb09c685ec3
* udev-rules: fix nvme symlink creation on namespace changes (LP: #2028180)
File:
debian/patches/lp2028180-udev-rules-fix-nvme-symlink-creation-on-namespace-changes.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=26e85b944da9098e66fc0c39f64ee40254c0c278
* core: reorder systemd arguments on reexec (LP: #2013543)
File:
debian/patches/lp2013543-core-reorder-systemd-arguments-on-reexec.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=19ba0f20d311642596dc65fa5d6eb96a2d4be280
* network/dhcp4: do not ignore the gateway even if the destination is in same
network (LP: #2009743)
Files:
-
debian/patches/lp2009743/network-dhcp4-do-not-ignore-the-gateway-even-if-the-desti.patch
-
debian/patches/lp2009743/test-network-add-one-more-testcase-for-DHCPv4-classless-r.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=73e774a2fb99c82df6d0edd770bb84ab735ec2f0
* Drop debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch (LP:
#1982218)
File:
debian/patches/debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=cf82f08feea456e1c65895b34bffa8c33d421588
* debian/systemd.postint: do not daemon re-exec if we could hit LP: #2013543
File: debian/systemd.postinst
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=be484dab06d590b1792a8f016f4292373d0174b7
* binfmt: fix systemd-binfmt in LXD containers (LP: #1999275)
Files:
-
debian/patches/lp1999275/binfmt-check-if-binfmt-is-mounted-before-applying-rules.patch
-
debian/patches/lp1999275/binfmt-util-also-check-if-binfmt-is-mounted-in-read-write.patch
- debian/patches/lp1999275/binfmt-util-split-out-binfmt_mounted.patch
-
debian/patches/lp1999275/unit-check-more-specific-path-to-be-written-by-systemd-bi.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0b9eadf05ee6db0e009dacbc71521480095880b3
[ Dan Streetman ]
* Fix machinectl pull-tar and import-tar (LP: #1977630)
Author: Dan Streetman
File: debian/patches/lp1977630-fix_machinectl_pull_tar.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1e7d3febe1600c6eb03bd71a17be6a6af52988c7
* make machinectl read-only work (LP: #1991829)
Author: Dan Streetman
File:
debian/patches/lp1991829-add-CAP_LINUX_IMMUTABLE-to-systemd-machined-so-it-ca.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7a7c47569e0a2d175915eb0b79c60f2611848731
-- Nick Rosbrook <en...@ubuntu.com> Mon, 21 Aug 2023 17:11:40 -0400
** Changed in: systemd (Ubuntu Jammy)
Status: Triaged => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/2013543
Title:
systemctl daemon-reexec forgets running services and starts everything
new
Status in systemd:
Fix Released
Status in systemd package in Ubuntu:
Fix Released
Status in systemd source package in Jammy:
Fix Released
Status in systemd source package in Lunar:
Fix Released
Bug description:
[Impact]
Depending on the contents of /proc/cmdline, when systemd is re-
executed with systemctl daemon-reexec, the --deserialize flag may be
ignored because it was added after the other arguments. For example,
if /proc/cmdline contains ---, then the re-exec cmdline might look
like:
$ cat /proc/1/cmdline | tr '\0' '\n'
/lib/systemd/systemd
---
splash
--system
--deserialize
54
This causes systemd not to process the --deserialize 54 argument,
causing it to start with a fresh state. This can cause all kinds of
problems, and one easy symptom to see is many lines in the journal
like:
"$service.service: Found left-over process $pid ($service) in control
group while starting unit. Ignoring."
[Test Plan]
1. (Only needed if your test system is not already affected) Edit the
kernel command line to contain '---' at the end, which would trigger
the bug. This can be done by appending '---' to
GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, running update-grub,
and then rebooting.
2. After enabling -proposed, install systemd:
$ apt install systemd -y
3. Check that the systemd.postinst script skipped the daemon-reexec
call, and instead indicated a reboot is required:
$ grep -Fsx systemd /run/reboot-required.pkgs
systemd
4. Reboot.
5. Try to re-exec systemd, and check that there are not tons of "left-
over process" log messages:
$ systemctl daemon-reexec
$ journalctl --grep "Found left-over process" -b 0
6. Also confirm that the ordering of /proc/1/cmdline is correct, i.e.
that --deserialize $fd comes before args from /proc/cmdline:
$ cat /proc/1/cmdline | tr '\0' '\n'
[Where problems could occur]
There are two changes for this bug. First is the patch against systemd
itself, which changes the ordering of arguments on the systemd
commandline. This change simply makes it so that systemd's own
arguments are always put first on it's re-exec commandline, and that
anything from /proc/cmdline is appended after. Any regressions caused
by this would also be seen in systemctl daemon-reexec invocations.
The second change is in systemd.postinst, which skips the systemctl
daemon-reexec call when upgrading from versions of systemd that could
hit this bug. Regressions caused by this would be seen during package
upgrades.
[Original Description]
# Our problem #
During a regular update of our container environment, `systemd` (and
the related packages libpam-systemd, libsystemd0, libudev1, systemd-
sysv and udev) were updated from `249.11-0ubuntu3.6` to
`249.11-0ubuntu3.7`. We're talking only about Ubuntu 22.04. Our Ubuntu
20.04 is working fine with `systemctl daemon-reexec`.
In my opinion, the update was not the problem because we've tried
downgrading and tried these versions: (current) `249.11-0ubuntu3.7`,
`249.11-0ubuntu3.6`, `249.11-0ubuntu3.4` and `249.11-0ubuntu3.3`. The
symptoms were the same.
# Symptoms #
The `/var/lib/dpkg/info/systemd.postinst` executes a `systemctl
daemon-reexec` and that ended in a disaster. It seems that `systemd`
is forgetting all it started children and tries to start nearly every
configured service again. Naturally, the old services are still
running, and the ports can't be opened twice and `systemd` won't give
up. Here are some(!) of the logfiles:
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Create Volatile Files and
Directories...
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found
left-over process 130 (systemd-udevd) in control group while starting unit.
Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean
termination of a previous run, or service implementation deficiencies.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found
left-over process 31475 (systemd-udevd) in control group while starting unit.
Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean
termination of a previous run, or service implementation deficiencies.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found
left-over process 31476 (systemd-udevd) in control group while starting unit.
Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean
termination of a previous run, or service implementation deficiencies.
And...
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target System
Initialization.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt download
activities.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt upgrade and clean
activities.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily dpkg database backup
timer.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Periodic ext4 Online
Metadata Check for All Filesystems.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Condition check resulted in Discard
unused blocks once a week being skipped.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily rotation of log files.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily man-db regeneration.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Message of the Day.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Clean PHP session files
every 30 mins.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Update the plocate database
daily.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily Cleanup of Temporary
Directories.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Basic System.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: System is tainted: cgroupsv1
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Timer Units.
And...
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Found left-over
process 206 (atd) in control group while starting unit. Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean
termination of a previous run, or service implementation deficiencies.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Deferred execution
scheduler...
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: cron.service: Found left-over
process 164 (cron) in control group while starting unit. Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean
termination of a previous run, or service implementation deficiencies.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Regular background program
processing daemon.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: dbus.service: Found left-over
process 177 (dbus-daemon) in control group while starting unit. Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean
termination of a previous run, or service implementation deficiencies.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started D-Bus System Message Bus.
And...
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: rsyslog.service: Found left-over
process 204 (rsyslogd) in control group while starting unit. Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Failed with result
'exit-code'.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Unit process 206 (atd)
remains running after unit stopped.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: apache2.service: Found left-over
process 382 (apache2) in control group while starting unit. Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean
termination of a previous run, or service implementation deficiencies.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: apache2.service: Found left-over
process 392 (apache2) in control group while starting unit. Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean
termination of a previous run, or service implementation deficiencies.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: apache2.service: Found left-over
process 397 (apache2) in control group while starting unit. Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean
termination of a previous run, or service implementation deficiencies.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: apache2.service: Found left-over
process 3052 (apache2) in control group while starting unit. Ignoring.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean
termination of a previous run, or service implementation deficiencies.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting The Apache HTTP Server...
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Stopped Deferred execution
scheduler.
Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Found left-over
process 206 (atd) in control group while starting unit. Ignoring.
And...
Mar 31 12:51:40 FQDN_REDACTED sshd[31772]: error: Bind to port 22 on
0.0.0.0 failed: Address already in use.
And...
Mar 31 12:52:06 FQDN_REDACTED systemd[1]: Started The Salt Minion.
Mar 31 12:52:06 FQDN_REDACTED salt-minion[32339]: The Salt Minion is shutdown.
Mar 31 12:52:11 FQDN_REDACTED systemd[1]: salt-minion.service: Main process
exited, code=exited, status=1/FAILURE
Mar 31 12:52:11 FQDN_REDACTED systemd[1]: salt-minion.service: Failed with
result 'exit-code'.
Mar 31 12:52:11 FQDN_REDACTED systemd[1]: salt-minion.service: Unit process
2808 (/opt/saltstack/) remains running after unit stopped.
Mar 31 12:52:11 FQDN_REDACTED systemd[1]: salt-minion.service: Unit process
2848 (/opt/saltstack/) remains running after unit stopped.
Other internal `systemd` process were started again:
root 1 0.0 0.1 101204 12444 ? Ss 10:19 0:03
/lib/systemd/systemd -z --system --deserialize 16
root 75 0.0 0.1 31440 13484 ? Ss 10:19 0:00
/lib/systemd/systemd-journald
systemd+ 159 0.0 0.0 16124 8004 ? Ss 10:19 0:00
/lib/systemd/systemd-networkd
message+ 177 0.0 0.0 8252 4440 ? Ss 10:19 0:00
@dbus-daemon --system --address=systemd: --nofork --nopidfile
--systemd-activation --syslog-only
root 205 0.0 0.0 14908 6464 ? Ss 10:19 0:00
/lib/systemd/systemd-logind
systemd+ 223 0.0 0.1 25268 12592 ? Ss 10:19 0:00
/lib/systemd/systemd-resolved
root 31424 0.0 0.1 31424 13636 ? Ss 12:51 0:00
/lib/systemd/systemd-journald
systemd+ 31636 0.0 0.0 16124 6588 ? Ss 12:51 0:00
/lib/systemd/systemd-networkd
message+ 31639 0.0 0.0 8124 3804 ? Ss 12:51 0:00
@dbus-daemon --system --address=systemd: --nofork --nopidfile
--systemd-activation --syslog-only
root 31682 0.0 0.0 14908 6480 ? Ss 12:51 0:00
/lib/systemd/systemd-logind
systemd+ 31686 0.0 0.1 25268 12580 ? Ss 12:51 0:00
/lib/systemd/systemd-resolved
root 32087 0.0 0.0 21436 5252 ? Ss 12:51 0:00
/lib/systemd/systemd-udevd
You can either kill all the old processes and restart them, and then
everything is fine. Or you can reboot the container. Besides that
`systemctl daemon-reexec` the `systemd` version is running fine.
`systemctl daemon-reload` is working like a charme.
# Normal case #
In the normal case a `systemctl daemon-reexec` just prints only a few
lines:
Mar 31 14:21:58 FQDN_REDACTED systemd[1]: Reexecuting.
Mar 31 14:21:58 FQDN_REDACTED systemd[1]: systemd 249.11-0ubuntu3.7 running
in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT
+GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD
+LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ
+ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Mar 31 14:21:58 FQDN_REDACTED systemd[1]: Detected architecture x86-64.
# Testcase #
Doing a `systemctl daemon-restart` and `ssh localhost` shows the
problem. `systemctl` removes the directory `/run/sshd` during the
reexec and `ssh` will refuse further connects because the directory is
missing.
$ systemctl daemon-restart
$ ssh root@localhost
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 22
$
Killing the old instance of SSH and restarting it will work.
# Some details to the hardware #
Our metal runs OpenVZ/Virtuozzo with this kernel (without any
problems):
> Linux FQDN_REDACTED 3.10.0-1127.18.2.vz7.163.46 #1 SMP Fri Nov 20
21:47:55 MSK 2020 x86_64 x86_64 x86_64 GNU/Linux
The container with the `systemctl daemon-reexec` problem reports the
following kernel:
Linux FQDN_REDACTED 5.4.0 #1 SMP Thu Apr 22 16:18:59 MSK 2021 x86_64
x86_64 x86_64 GNU/Linux
# Upshot #
* Can somebody help me with this issue?
* Why is `systemctl` losing its internal state about the running
processes/services?
* Why is `systemctl` restarting everything?
To manage notifications about this bug go to:
https://bugs.launchpad.net/systemd/+bug/2013543/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
