Hi Simon, if you are short on time, you might skip right to the end of this mail and ignore my other comments as they are not of any high priority since at least for me, this issue is solved with somehow integrating debvm or adding an interface that allows using debvm output with autopkgtest.
Quoting Simon McVittie (2022-06-08 11:57:58) > On Sun, 05 Jun 2022 at 17:04:02 +0200, Johannes Schauer Marin Rodrigues wrote: > > I am offering to either rewrite autopkgtest-build-qemu with a --method > > option > > defaulting either to "guestfish" if a native image is requested, > > "initramfs" if > > a foreign architecture is requested and "kernel" if the image is neither > > i386, > > amd64, armhf, arm64 or ppc64el. If the user requires a system as close to a > > real system as possible, they can manually use --method=debian-installer. > > This is extremely comprehensive, but I'm concerned that if we have this > many code paths for building a qemu VM, at least one of them will be > broken at any given time. We already have more virtualization backends > than maintainer bandwidth or routine testing (see my recent efforts to > get all our tests passing, and I haven't even got as far as qemu yet). > > I appreciate your offer to add more code, but if we add all these code > paths and document them as something that should work, I'm concerned that > we'll end up in a situation where we can't remove any of them because > someone might be relying on them, but we don't actually know that they > *work* either, and nobody among the autopkgtest maintainers has the time > or knowledge to fix them when they don't. (This could be mitigated by > outsourcing as much as possible to some other system builder, like we > already do to some extent with vmdb2.) those are very valid concerns. This makes sense. > As I believe I've already said in an earlier message to this bug report, > one way to get autopkgtest out of the business of being yet another > system-building tool is to start from a "cloud" VM image produced by > someone else (ideally, "have a qemu-runnable cloud image" would be one > of the jobs for the porting team for each architecture). One approach to > this is to take a generic cloud image, boot it with qemu and modify it > to provide the serial ports that autopkgtest-virt-qemu currently needs, > as autopkgtest-buildvm-ubuntu-cloud does. My concern with using downloadable cloud images is that what I download from some server is not verified by the archive GPG keys. I much rather run one of the existing system-building tools which only downloads data verified by debian-archive-keyring. But in another threat you already said that you'd like the input to autopkgutils just being a tarball created by some other tool and I'd be completely happy with that idea. > However, the serial-port-based command execution is really quite > fragile: there are arbitrary timeouts that cause testbed startup to > fail randomly while under load (which is a large part of why I haven't > been able to get it systematically passing tests), and it's vulnerable > to architectures having different conventions for how the virtualized > "hardware" is represented in the guest user-space. I've tried to reduce > that problem by using the virtio hypervisor serial consoles /dev/hvc*, but > as you can see from the ppc64le special case in lib/autopkgtest_qemu.py, > even that is not completely predictable, because on ppc64le the virtio > serial consoles start from hvc1. > > I think a less fragile approach than the serial ports would be to use > ssh, which in practice needs to work anyway, because it's how porters > typically interact with their test hardware. My goal in refactoring > part of autopkgtest-virt-qemu into lib/autopkgtest_qemu.py was to get an > autopkgtest-virt-ssh setup script that runs and manages a qemu VM, using > cloud-init to provision a trusted public key. Unfortunately, getting that > tested and working has been delayed by trying to clear enough technical > debt to make the tests pass on the backends we already have... Is the argument here, that you are not happy with how autopkgtest-virt-qemu operates in general and that's an additional reason (probably in connection with the maintenance burden) why you don't want to add new features to it? > I think the proliferation of system-building tools that you have noted in > the past is partly because until recently they have always needed root, > partly because none of the existing ones is particularly comprehensive, > and partly because of the number of fiddly details involved in getting > the correct bootloader for each architecture and poking it into place. > If we had something analogous to grub-cloud (install on and boot from the > first disk, no attempt to support dual-booting or special configurations, > hard-code a serial console, etc.) as part of the minimal requirements > for supporting an architecture, ideally with a metapackage available on > all architectures that pulls in the correct grub-cloud-equivalent, then I > think that would help a lot - but unfortunately grub-cloud is amd64-only. As far as I know, there is no bootloader that supports all our release architectures, no? > It also occurs to me that thanks to debootstrap and its clones (including > mmdebstrap), making a non-bootable Debian rootfs tree (in some transport > format like a tarball) is a solved problem; but one of the reasons people > have quality-of-implementation preferences between system-building tools > is that all of them try to implement the entire procedure, starting from > nothing and ending up with a bootable image. Might it help to re-scope > the problem to: start from a non-bootable rootfs tarball as produced by > debootstrap/mmdebstrap, and do whatever architecture-specific things > are necessary to turn it into a bootable disk image with a kernel, > a bootloader, whatever architecture-specific boot partitions are required, > and an expandable root partition populated from the tarball? Isn't that what many system-build-tools already do? They do take the debootstrap output and then turn that into a bootable disk but they don't re-invent debootstrap. In that sense, that is also what autopkgtest-build-qemu does. It takes the output of debootstrap (well, it calls debootstrap itself but could probably easily be modified to take an existing debootstrap tarball) and then adds that to a system image. > > debian-installer (option 3 from above) > > > > - useful because this creates a system that is as close to a system as users > > would install it as possible -- but it takes longer to create than the > > others > > Does this work on all architectures, or on all release architectures? No. I wasn't able to figure out some of them like mips* or s390x. > I think something more analogous to the official cloud images is perhaps > a better basis for CI systems than preseeding the interactive d-i > interface so much that it becomes non-interactive, but if this has wider > architecture coverage than (for example) the initramfs approach then it > seems maybe useful. The d-i approach unfortunately is also very slow. > > Option 4 is to use u-boot-qemu > > If we had u-boot-qemu on more architectures, I'd like this one a lot more. Yeah, me too. :) > > Option 5 is the same as above but will work for architectures without > > u-boot-qemu support. Without a bootloader, the kernel and initrd can be > > passed to qemu manually via -kernel and -initrd arguments. > > The major disadvantage of booting with an external kernel and initrd is > that it can't test new kernels, and can't fully test new versions of a > component that contributes to the initramfs, which might break the > expectations of a test that has the isolation-machine restriction. True, but there are different trade-offs here, I think. While it would of course be nice to have one qemu backend that is able to do everything, I only see multiple different variations that each have their advantages and disadvantages. While this method cannot test new kernels, it provides a wider architecture support that was not possible with existing solutions. > > vmdb2 needs root and this will not change > > mkosi needs root > > As soon as you have one qemu-bootable image (perhaps from adapting a > cloud image made by someone else), you can boot into that to be root > and produce others. On amd64, you don't even need an image, because > fakemachine can reuse your running system's kernel and /usr (but that's > currently amd64-only). Hm... it would still be nice to avoid having to double-layer this just because some tools decided that they just require superuser privileges even though they are not a technical necessity. > > debos will only run on amd64 and not on the other arches > > If I knew how to design APIs in Go, I'd have fixed that already: > it's not actually all that hard conceptually, but it does need a table > mapping Go architectures to various architecture-specific properties, > and I don't know how a Go programmer would represent that in a somewhat > future-proof API. If we all had more time, so much could be done. :D So this issue is about allowing autopkgtest-build-qemu to create a bootable image for autopkgtest-virt-qemu without superuser privileges. I did a lot of investigation into the problem and implemented some proof-of-concept solutions (see my earlier mails to this bugreport) but I wasn't completely happy with any of them because they were either slow, didn't cover all release arches, or were not reproducible. Now helmut has provided debvm which (as far as my needs go) covers all these issues. It's like my option 5 but uses debugfs to extract the kernel and initrd from the filesystem image. It thus has all the benefits of option 5 but ships the result in a single file and even allows upgrading the kernel. As far as I'm concerned, this issue can be closed once it somehow becomes possible to run autopkgtest via debvm: https://bugs.debian.org/1033291 https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/236 https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/237 Francesco, you initially reported this issue. Do you share my conclusion that debvm solves the problem you raised about creating autopktest disk images without superuser privileges? Thanks! cheers, josch
signature.asc
Description: signature
