Johannes, thank you for this enlightening update!
You're far ahead of any ideas I've been having (some of which I attribute to Franscesco), but I'd nevertheless like to contribute a few thoughts just in case. On 2022-03-23 12:43, Johannes Schauer Marin Rodrigues wrote: > Quoting Francesco Poli (2022-03-23 00:09:21) > Full: the problem with the current version of > mmdebstrap-autopkgtest-build-qemu > is, that it can only build qemu images for the native architecture. This is > because it relies on guestfish. Guestfish will never be able to operate on > foreign architecture qemu guests. This is because: > > - guestfish sets architecture specific options at compile time. This means > that every guestfish binary can only be used for qemu guests of the same > architecture as that binary and this cannot be changed at runtime > > - guestfish relies on another program called supermin. Essentially, supermin > assembles a minimal chroot which is then loaded as qemu boots and then > carries out the guestfish operations. Since supermin just copies binaries > from the host, it cannot create foreign chroots either. This is unfortunate, as otherwise guestfish would be universal, but it's understandable. I initially wondered whether foreign architecture guestfish/supermin binaries couldn't be run through qemu-user-static, but that solution would be just far too hacky I think. > This means we have to replace guestfish by something else. I'm not aware that > this already exists so I wrote a proof-of-concepts that does what we need. It > works by first building a kernel and initramfs and then booting qemu with > both. > The initramfs contains scripts that partition the disk, copy the rootfs and > install the bootloader: > > main script: http://paste.debian.net/1235312/ > initramfs-hook: http://paste.debian.net/1235313/ > initramfs-script: http://paste.debian.net/1235314/ This is really neat! > This works for both, foreign and native architectures. But since running > mmdebstrap to build the initramfs is far slower than supermin, I'd like to > fallback to using guestfish in the native case. So I have to combine above > script with mmdebstrap-autopkgtest-build-qemu. > > I'm yet unsure whether I want to make these more general so that they can be > used for other purposes or whether the script should specifically build > autopkgtest qemu images. > > A disadvantage is, that this only works for architectures for which qemu knows > how to boot them without kernel and initrd from the outside. This means that > mips* and s390x are not supported and neither are most of the Debian ports > architectures. I don't know how to solve this other than by teaching > autopkgtest-virt-qemu that now it needs three input files: the kernel, the > initrd and the rootfs. Since these are QEMU options, I think autopkgtest-virt-qemu's existing --qemu-options could be used to supply kernel/initrd/rootfs? > Another disadvantage is, that the output can never be bit-by-bit reproducible > because grub-install is unreproducible. > > I've worked on this in context with replacing vmdb2 in autopkgtest-build-qemu > so that sbuild-qemu-create (maintained by Christian Kastner) can be run > without > superuser privileges. We've tried to approach the vmdb2 author but Lars is > reluctant to include such drastic changes: > https://gitlab.com/larswirzenius/vmdb2/-/issues/62 > > So I'll probably release yet another https://wiki.debian.org/SystemBuildTools > because the existing solutions don't do what I want. The closest is probably > debos which can be run without being root because everything is done inside > qemu. But it also doesn't solve the hard problem of installing a bootloader, > leaving it to the user: https://github.com/go-debos/debos/issues/137 I think this is a fantastic idea, not just because sbuild-qemu would be a major beneficiary but because there is a general gap here: there currently is no tool that can easily create images (1) without root and (2) for arbitrary architectures. Having such a tool would enable many upstreams to easily test their software on other architectures, e.g. via GitHub Actions. PowerPC and RISC-V are interesting contenders but only the fewest projects test them. However, our buildds and debci discover issues all the time. Additionally, it's really useful to have an on-demand, throwaway porterbox when you need one. I probably use sbuild-qemu-boot more than sbuild-qemu, and the former is just a QEMU launcher. I use it even for native architectures because it's basically a container-like workspace where I can do Bad Stuff without fear of polluting or trashing my system. I've run out of ideas how to easily solve this, other than what Francesco suggested (e.g. create a base with FAI or preseed.cfg, then finalize by running commands in the VM just as sbuild-qemu-update does it). Really looking forward to what you are working on. If you need a test user, I'll happily volunteer! Best, Christian
