[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2024-05-30 Thread Nick Rosbrook
** Changed in: systemd (Ubuntu) Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails to activate mount units in Ubuntu Core 18 To mana

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-19 Thread Michael Vogt
Fwiw, it looks like the latest core18 edge build that includes https://launchpad.net/ubuntu/+source/cloud-init/21.4-0ubuntu1~18.04.1 make the problem go away. It seems https://bugs.launchpad.net/cloud- init/+bug/1946003 is what caused it. The old code had a udev rule that would call into the cloud

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-19 Thread Dan Streetman
> It's not something that snapd writes out, it's curious where this comes from it's not written out, it's internally handled/added by pid1, and it *should* be there, because you do want (for example) the mount to be stopped if its backing device is removed. However when the mount is stopped, the B

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-19 Thread Michael Vogt
I attached the /run/cloud-init and /var/log/cloud-init logs of the good/bad run - I looked over the diff via "diff -u <(cut -f 4- -d: /tmp/cloud-init-good-core18-r2206/cloud-init.log) <(cut -f 4- -d: /tmp/cloud-init-bad-core18-r2208/cloud-init.log)" but couldn't see anything standing out there (but

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-19 Thread Michael Vogt
** Attachment added: "cloud-init logs for the "good" run" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1949089/+attachment/5542024/+files/cloud-init-good-core18-r2206.tar.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-19 Thread Michael Vogt
** Attachment added: "cloud-init logs for the "bad" run" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1949089/+attachment/5542025/+files/cloud-init-bad-core18-r2208.tar.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. h

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-18 Thread Michael Vogt
I spend a bit more time bisecting the various core18 snaps to see at what revision the failure started to become reproducible. Interestingly I found that it started with r2208 AFAICT. It's all a bit annoying because it's a race so I don't trust it 100% but the previous r2206 got 2 good runs in a ro

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-18 Thread Michael Vogt
@Dan Thanks, this find about BindsTo is super interesting! It's not something that snapd writes out, it's curious where this comes from. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title:

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-18 Thread Dan Streetman
This added BindsTo is what's causing the problem: ddstreet@localhost:~$ systemctl show -p BindsTo snap-hello\\x2dworld-29.mount BindsTo= ddstreet@localhost:~$ sudo systemctl daemon-reload ddstreet@localhost:~$ systemctl show -p BindsTo snap-hello\\x2dworld-29.mount BindsTo=dev-loop6.device ubu

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-18 Thread Dan Streetman
Sorry to clarify that, the dep is added to the mount over a daemon- reload, but only on core: ddstreet@localhost:~$ systemctl list-dependencies snap-hello\\x2dworld-29.mount snap-hello\x2dworld-29.mount ● ├─-.mount ● ├─snap.mount ● ├─system.slice ● └─var-lib-snapd.mount ddstreet@localhost:~$ syst

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-18 Thread Dan Streetman
Note one interesting point is that in a core18 system, a squashfs mount (any loop-based mount, really) includes a dep from the mount to its loop device, while the non-core system does not: ddstreet@localhost:~$ cat /etc/issue Ubuntu Core 18 on \4 (\l) ddstreet@localhost:~$ systemctl list-dependen

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-18 Thread Lukas Märdian
Thanks ddstreet, I agree 918e6f1c0151429f5095355f4f3f74f16e79724a could be related as well, but unfortunately that commit produced quite some fall-out. Nevertheless, I've prepared a new systemd build in my PPA and the "core18_2028_amd64.snap" at https://people.ubuntu.com/~slyon/uc18/ cherry-pi

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-17 Thread Dan Streetman
> @slyon you might want to see if commit 918e6f1c0151429f5095355f4f3f74f16e79724a fixes this (and there are a couple follow on commits to this) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 T

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-17 Thread Michael Vogt
I spend a bit of quality time with this bug today and it seems there is (also?) a kernel dimension to it. I ran the following: """ $ git status On branch tests-use-core18-from-gce Your branch is up to date with 'sergiocazzolato/tests-use-core18-from-gce'. $ git diff diff --git a/overlord/devicesta

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-17 Thread Dan Streetman
@slyon you might want to see if commit 918e6f1c0151429f5095355f4f3f74f16e79724a fixes this -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails to activate mount units

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-16 Thread Alberto Mardegan
Thanks Lukas! It's very likely that there are several issues in this area, because while I can confirm that your latest image fixes the reproducer I pasted before (in the Google cloud, too), the spread test failure is still reproducible with the "tests/main/services-disabled- kept-happy" test. I n

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-15 Thread Lukas Märdian
Thank you, having a reliable reproducer is very helpful! I can confirm this reproduces locally for me on QEMU. I prepared a new systemd package [0], containing backports from the above mentioned PR that indeed sounds very related (I wonder why the issue didn't happen before, tho, as this is unrela

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-14 Thread Alberto Mardegan
One more update: I verified that the steps from comment #17 can be used to reproduce the issue in an Ubuntu Core image started in qemu, and without using the spread. I verified it with the 20211105 image provided by Lukas in https://people.ubuntu.com/~slyon/uc18/ To reproduce it, start the image i

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-11 Thread Alberto Mardegan
Might be related to https://github.com/systemd/systemd/issues/11342 ** Bug watch added: github.com/systemd/systemd/issues #11342 https://github.com/systemd/systemd/issues/11342 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https:

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-11 Thread Alberto Mardegan
An update to the comment above. We have found out a sequence of commands that seems to reproduce the issue reliably: in the spread session started by the command from the comment above, if you type this: systemctl stop 'snap-disabled\x2dsvcs\x2dkept-x1.mount' systemctl daemon-reload #

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-11 Thread Alberto Mardegan
> can you provide steps on how this is reproducable then? So far we have not been able to isolate the problem, so the only way so far is to run the spread tests from the snapd directory (from the master branch): spread -debug -repeat 20 google:ubuntu- core-18-64:tests/main/services-disabled-k

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-09 Thread Dan Streetman
> and in QEMU it seems that this bug is not reproducible can you provide steps on how this is reproducable then? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-09 Thread Alberto Mardegan
I'm coming with bad news. Today both me and Michael have individually retested both snaps you provided (20211102 and 20211105), and the bug happened with both of them. It's possible that yesterday Michael accidentally used QEMU for his tests, without realizing (and in QEMU it seems that this bug is

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-08 Thread Lukas Märdian
** No longer affects: systemd (Ubuntu Jammy) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails to activate mount units in Ubuntu Core 18 To manage notifications ab

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-08 Thread Michael Vogt
About the question about Focal/hirsute/uc20 - we have not observed the issue there. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails to activate mount units in Ubu

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-08 Thread Lukas Märdian
That's certainly possible. In the first attempt (that failed according to mvo's tests) I only backported https://github.com/systemd/systemd-stable/commit/04eb582acc203eab0bc5c2cc5e13986f16e09df0 and https://github.com/systemd/systemd-stable/commit/cc6271f17d3c5c200e8a391f10d0afb71403fc28 (as a de

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-08 Thread Michael Vogt
Fwiw, the patch debian/lp1934147/0001-core-add-a-new-unit-method- catchup.patch is part of a fairly large patchset (https://github.com/systemd/systemd/pull/9200/commits) so maybe something is missing there? -- You received this bug notification because you are a member of Ubuntu Bugs, which is su

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-08 Thread Lukas Märdian
** Merge proposal linked: https://code.launchpad.net/~slyon/ubuntu/+source/systemd/+git/systemd/+merge/411528 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-08 Thread Lukas Märdian
Thank you for testing the revert! Do we know if the same issue happens in Focal/UC20 as well? Or can we somehow falsify that? The regressing patches had been introduced to Hirsute/Focal/Bionic to fix LP: #1934147 -- You received this bug notification because you are a member of Ubuntu Bugs, whi

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-08 Thread Łukasz Zemczak
Thank you Michael for verifying this! As per my earlier question during our last-week's sync: do you think it would be possible to have this running for instance as part of the coreXX-edge testing that Sergio is doing? Since we'd like to make sure we don't reintroduce this issue at some point by ac

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-08 Thread Michael Vogt
Just for the record, I also did another run with: https://storage.googleapis.com/snapd-spread- tests/snaps/core18_20211102_amd64.snap and that also fails right away: """ $ git status On branch tests-use-core18-from-gce Your branch is up to date with 'sergiocazzolato/tests-use-core18-from-gce'. $ g

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-08 Thread Michael Vogt
Thanks Lukas for providing this revert of f0831ed2a03fcef582660be1c3b1a9f3e267e656. Using https://people.ubuntu.com/~slyon/uc18/core18_20211105_amd64.snap I can no longer reproduce the isue. To make it easier to reproduce what I did: """ $ git status On branch tests-use-core18-from-gce Your branch

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-05 Thread Łukasz Zemczak
** Changed in: systemd (Ubuntu Bionic) Assignee: (unassigned) => Lukas Märdian (slyon) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails to activate mount unit

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-05 Thread Lukas Märdian
Sounds like we cannot have an easy fix by simply applying 04eb582acc203eab0bc5c2cc5e13986f16e09df0 (and its dependencies). So let's first try to pinpoint if f0831ed2a03fcef582660be1c3b1a9f3e267e656 really introduced the regression. I've built a new core18 snap with my previous work and the inclu

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-05 Thread Lukas Märdian
** Also affects: systemd (Ubuntu Bionic) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails to activate mount units in Ub

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-05 Thread Michael Vogt
We created a test PR that uses this core18 snap build in https://github.com/snapcore/snapd/pull/11015 - we are still running tests but it seems the error is still there :-( -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.l

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-04 Thread Brian Murray
** Tags removed: rls-jj-incoming ** Also affects: systemd (Ubuntu Jammy) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fa

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-04 Thread Matthieu Clemenceau
** Tags added: fr-1832 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails to activate mount units in Ubuntu Core 18 To manage notifications about this bug go to: ht

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-04 Thread Lukas Märdian
Sure, I put that custom core18 snap into https://people.ubuntu.com/~slyon/uc18/ as well. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails to activate mount units i

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-03 Thread Ian Johnson
@slyon could you make available the core18 snap you built with this systemd ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails to activate mount units in Ubuntu Co

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-11-02 Thread Lukas Märdian
Not having a reliable reproducer is always a tricky situation to be in... Indeed that other commit might be related. I've backported the following patches and created a custom systemd build in https://launchpad.net/~slyon/+archive/ubuntu/lp1949089 https://github.com/systemd/systemd-stable/commit/

[Bug 1949089] Re: systemd randomly fails to activate mount units in Ubuntu Core 18

2021-10-28 Thread Brian Murray
** Tags added: rls-jj-incoming -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1949089 Title: systemd randomly fails to activate mount units in Ubuntu Core 18 To manage notifications about this bug g