** No longer affects: linux-azure (Ubuntu)
** No longer affects: systemd (Ubuntu)
** Also affects: cloud-utils (Ubuntu Eoan)
Importance: Undecided
Status: New
** Also affects: cloud-initramfs-tools (Ubuntu Eoan)
Importance: Undecided
Status: New
--
You received this bug not
Yes, we see it on Eoan and blocking tests, so if you could SRU it there,
it'd be much appreciated. We have never seen it on Bionic to my
knowledge.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launch
Great, thanks for confirming.
I wonder if we should SRU this. You see it on Eoan I presume? The bug is
theoretically present in Bionic too aiui but if it never crops up I
don't know if it's worth it.
--
You received this bug notification because you are a member of Kernel
Packages, which is subs
Sorry, we were waiting on a bug regarding dictionary keys being modified
to resolve before we could verify this. We have not seen this in the
past couple of runs in our testing, so I think we're good to proceed.
--
You received this bug notification because you are a member of Kernel
Packages, w
Can someone from CPC confirm that this has fixed the issues in the Azure
tests?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
St
This bug was fixed in the package cloud-utils -
0.31-7-gd99b2d76-0ubuntu1
---
cloud-utils (0.31-7-gd99b2d76-0ubuntu1) focal; urgency=medium
* New upstream snapshot.
- growpart: add flock support to prevent udev races (LP: #1834875)
-- Ryan Harper Tue, 25 Feb 2020 14:41:21
-0
This bug was fixed in the package cloud-initramfs-tools - 0.45ubuntu1
---
cloud-initramfs-tools (0.45ubuntu1) focal; urgency=medium
* Add dependency on flock for growroot's use of growpart.
(LP: #1834875)
-- Scott Moser Tue, 25 Feb 2020 13:08:22 -0500
** Changed in: cloud-i
On Tue, Feb 25, 2020 at 2:35 PM Scott Moser
wrote:
> this seemed to "just work" for me.
> http://paste.ubuntu.com/p/93dWDPZfZT/
Ah, I didn't check that there was an existing ubuntu/devel branch. Sorry.
I've pushed a MR here:
https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/
** Merge proposal linked:
https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/+merge/379851
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-in
this seemed to "just work" for me.
http://paste.ubuntu.com/p/93dWDPZfZT/
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
Status in
** Attachment added: "tarball of source package to upload"
https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5330895/+files/cloud-utils_0.31-7-gd99b2d76-source.tar.xz
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-a
@Scott,
cloud-utils isn't quite new-upstream-snapshot out of the box; the debian
dir does not contain the changelog; however, I think I've got this
sorted out. I've a MP I can put up; but it only will show the add of
the changelog file. I'll attach a debdiff and a source package.
--
You recei
** Patch added: "debdiff showing the changes to upload to fix the bug."
https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5330894/+files/cloud-utils_0.31-6_to_0.31-7.debdiff
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to
a.) I gave the wrong link. ugh. It should have been:
https://code.launchpad.net/~smoser/cloud-initramfs-tools/+git/cloud-initramfs-tools/+merge/379774
b.) the fixed link to 'a' probably makes more sense now. But basically
you need a newer cloud-initramfs-tools to adjust for the fact that
growp
@smoser
a looks merged
b has not had any changes since 2018 and looks up-to-date in Focal. What
am I missing?
c I see a devel branch, is it assumed that it uses a packaging structure
similar to cloud-init or can someone just grab and upload master?
--
You received this bug notification because
The fix is in cloud-utils upstream now.
Still to do:
a.) review/merge cloud-initramfs-tools pull request
https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/+merge/379177
b.) upload cloud-initramfs-tools to focal
c.) upload cloud-utils to focal
d.) any SRU
the order of 'b' a
** Changed in: cloud-initramfs-tools (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
Stat
** Merge proposal linked:
https://code.launchpad.net/~smoser/cloud-initramfs-tools/+git/cloud-initramfs-tools/+merge/379774
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
** Merge proposal linked:
https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/+merge/379177
** Changed in: cloud-utils
Status: New => Fix Committed
** Also affects: cloud-utils (Ubuntu)
Importance: Undecided
Status: New
** Changed in: cloud-utils (Ubuntu)
Here's the upstream changes to growpart I'm suggesting:
https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-
utils/+merge/379177
I've also proposed on modifications to cloud-init's cc_growpart as a further
method
to aid debugging if this hit as well as some mitigation around the race.
h
** Merge proposal linked:
https://code.launchpad.net/~raharper/ubuntu/+source/cloud-utils/+git/cloud-utils/+merge/378839
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Ti
Yes, this is my read on the issue as well. The trigger is related to
the inotify watch that systemd-udevd puts on the disk. Something that
might help that we could try per xnox's comment around use of flock.
if growpart were to flock /dev/sda (we need to sort out what flags are
needed to prevent
> Ah, no I think this might be along right lines: udev is calling blkid
> on the _partition_ of course, so it can probe for filesystem etc without
> looking at the partition table. After it's done that, it does look for
> the partition table so it can read the ID_PART_ENTRY_* values from it,
> but
Adding in a passing version of journalctl
** Attachment added: "passing_journalctl_output.txt"
https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5326024/+files/passing_journalctl_output.txt
--
You received this bug notification because you are a member of Kernel
Packages, which
Cloud-init service starts and will run growpart, etc
Feb 06 00:37:26 ubuntu systemd[1]: Starting Initial cloud-init job
(pre-networking)...
Feb 06 00:37:37 test-xrdpdnvfctsofyygmzan systemd[1]: Starting Initial
cloud-init job (metadata service crawler)...
Something has modified sdb1 (growpart/s
I'm attaching the output of a failing build
** Attachment added: "failing_journalctl_output.txt"
https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5325941/+files/failing_journalctl_output.txt
--
You received this bug notification because you are a member of Kernel
Packages, whic
On another tangent, I wonder if the change that brought this to light
was the switch to booting without initrd. The timing is about right, and
fits with the fact that it doesn't occur with the generic kernel (which
cannot boot without initrd in Azure). So if someone has an excess of
time to test ,
I can get that going, and will report back on this bug when I have more
information.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with ude
Hi, could someone prepare an image with
https://paste.ubuntu.com/p/HYGv6m8gCc/ at /etc/systemd/system/systemd-
udevd.service.d/udevd-debugging.conf, boot it on azure until it fails
and put the journalctl output (probably a few megs) somewhere I can read
it? Output from a successful boot would also
Oh yeah and one other thing I don't understand: why udev is processing
the partition while sgdisk is still running.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cl
Ah, no I think this might be along right lines: udev is calling blkid on
the _partition_ of course, so it can probe for filesystem etc without
looking at the partition table. After it's done that, it does look for
the partition table so it can read the ID_PART_ENTRY_* values from it,
but if it fail
So I've been handed the job of moving this bug forward this iteration.
Having just re-read all the comments again and read some source code
while listening to loud techno music my observations/questions are
these:
1) this is a doozy
2) I would really like to see complete udevadm monitor and inot
On Thu, 7 Nov 2019 at 20:05, Scott Moser wrote:
>
> > > So that means we have this sequence of events:
> > > a.) growpart change partition table
> > > b.) growpart call partx
> > > c.) udev created and events being processed
>
> > That is not true. whilst sfdisk is deleting, creating, finishing
On Thu, 7 Nov 2019 at 17:50, Ryan Harper <1834...@bugs.launchpad.net> wrote:
>
> @ddstreet
>
> Yes, settle does not help.
>
> Re-triggering udevadm trigger --action=add /sys/class/block/sda
>
> Would re-run all of them after the partition change has occurred, which
> is what I'm currently suggestin
> We can't know how many devices are in the system. It maybe nearly zero,
> it could be minutes.
I'm not suggesting you trigger uevents for all devices, just the one
you're repartitioning...
> The cold plug is required (initramfs may or maynot
> have ran
> scripts/udevd etc but kernel events alre
On Thu, Nov 7, 2019 at 2:41 PM Dan Streetman
wrote:
> > Issuing a second
> > trigger will repeat this.
> > IMO, that's a non-zero amount of time that slows the boot down, so I'd
> like
> > to avoid that.
>
> systemd-udev-trigger.serivce retriggers *everything* at boot (except in
>
an unprivileged
> Issuing a second
> trigger will repeat this.
> IMO, that's a non-zero amount of time that slows the boot down, so I'd like
> to avoid that.
systemd-udev-trigger.serivce retriggers *everything* at boot (except in
an unprivileged container where it can't), so I'm not sure how much
added time we're
On Thu, Nov 7, 2019 at 11:30 AM Dimitri John Ledkov
wrote:
> > So that means we have this sequence of events:
> > a.) growpart change partition table
> > b.) growpart call partx
> > c.) udev created and events being processed
>
> That is not true. whilst sfdisk is deleting, creating, finishing
> > So that means we have this sequence of events:
> > a.) growpart change partition table
> > b.) growpart call partx
> > c.) udev created and events being processed
> That is not true. whilst sfdisk is deleting, creating, finishing
> partition table (a) and partx is called (b), udev events ar
On Thu, Nov 7, 2019 at 1:30 PM Dan Streetman
wrote:
> > Yes, settle does not help.
>
> Well, I didn't suggest just to settle ;-)
>
Sorry; long bug thread.
> > I'm currently suggesting as a heavy-handed workaround.
>
> I don't really see why you think this is heavy-handed, but I must be
> missi
> Yes, settle does not help.
Well, I didn't suggest just to settle ;-)
> I'm currently suggesting as a heavy-handed workaround.
I don't really see why you think this is heavy-handed, but I must be
missing something.
> I would like to understand *why* the udevd/kernel pair exhibits this racy
>
@ddstreet
Yes, settle does not help.
Re-triggering udevadm trigger --action=add /sys/class/block/sda
Would re-run all of them after the partition change has occurred, which
is what I'm currently suggesting as a heavy-handed workaround.
I would like to understand *why* the udevd/kernel pair exhi
Just curious, did anyone actually test with my suggestion from comment
40? That is, just make your partition table change (and update the
kernel with partx or whatever, of course), and after it's done trigger
--settle udev for the device. Be interesting to know if that actually
fixes it or not.
> So that means we have this sequence of events:
> a.) growpart change partition table
> b.) growpart call partx
> c.) udev created and events being processed
That is not true. whilst sfdisk is deleting, creating, finishing
partition table (a) and partx is called (b), udev events are already
fi
I really think you are all *way* over thinking this.
a. growpart made a change to the partition table (using sfdisk)
b. growpart called partx --update --nr 3 /dev/sda
c. growpart exited
With a and b growpart created udev events. If you create udev events,
you really need to wait for those event
> it will prevent udevd from running the rules against it. Thus
effectively the event will be fired and done, but nothing actually
executed for it.
Interesting, I suspect this is the race we see. The events emitted but
no actions taken (ie we didn't get our by-partuuid symlink created.
> I someh
flock will not block inotify or udev events being emitted.
See
https://github.com/systemd/systemd/blob/master/src/udev/udevd.c#L322
https://github.com/systemd/systemd/blob/master/src/udev/udevd.c#L409
it will prevent udevd from running the rules against it. Thus
effectively the event will be fire
A couple of comments on the suggested path:
> Imho the sequency of commands should be:
> * take flock on the device, to neutralise udev
+1 on this approach. Do you know if the flock will block
systemd's inotify write watch on the block device which triggers
udevd? This is the typical race we se
@ Cloud init team, do we want to try changing cloud-utils to use a lock?
And like have a canary "only use locked codepath on this region" such
that we can assert through testing that this no longer happens with new
code, but does with old code.
--
You received this bug notification because you a
** Changed in: systemd (Ubuntu)
Status: New => Incomplete
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
Status in cloud-i
Some observations:
* growpart uses sfdisk without --no-tell-kernel option, meaning that it does
notify kernel about partition changes
* growpart later calls partx, which may be redundant / cause no changes or
events
* as a side note, partprobe, blockdev --rereadpt can also be used to reread
part
** Tags added: id-5dbb311b74e3f364d8f98c56
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
Status in cloud-init:
Incomplete
Stat
> The lack of events is not the issue
I'm not saying it is. But you seem to have it under control; I was just
offering my opinion.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/183
The lack of events is not the issue. We see all the events from the
kernel that we would expect to see during a resize, and we see the same
number (and type) of events on both the passing and the failing
instances.
The problem is that if udev processes the sda1 event first then, at
least some of
Oh, good observation, Dan. Yes, it was happening on Cosmic and later.
But I cannot say when it started. Now in a moment of despair I tried
something very unorthodox: I copied udevadm from Bionic to the Eoan
image and ran the tests again. Look and behold, all pass.
** No longer affects: linux-azure
(Oh, and thank you for that test run!)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
Status in cloud-init:
In Progress
Status
Now one wrinkle about it being a linux-azure bug: if you _aren't_ seeing
this on bionic but _were_ seeing it on cosmic, then there must be
something else involved, because I believe (based on the publishing
history[0]) that cosmic was on the same kernel version as bionic.
This could, of course, st
** Also affects: linux-azure (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1834875
Title:
cloud-init growpart race with udev
58 matches
Mail list logo
