Hi Josch, On 2025-01-02 13:21, Johannes Schauer Marin Rodrigues wrote: > Hi, > > Quoting Aurelien Jarno (2025-01-02 12:41:24) > > > Quoting Aurelien Jarno (2025-01-01 19:44:16) > > > > It is regularly claimed that mmdebstrap should be used instead of > > > > debootstrap. > > > how are the regular claims of some relevant to this report? > > This is to give some context about why I explored replacing debootstrap by > > mmdebstrap on the build daemons. > > hmmm... why would you do that? Is debootstrap doing something that you don't > like? Is mmdebstrap adding anything to the table that makes it superior to > debootstrap for that use-case?
In the long term, once we can get rid of bullseye and bookworm, or alternatively fix their package so that they all build in unshare mode, we will like to also generate the chroot in unshare mode. mmdebstrap is not the only solution, sbuild-createchroot can also just do that, and is closer than the current solution. OTOH mmdebstrap is closer to what is recommended for reproducing the build daemon setup. In any case testing it the build daemon context is always interesting to be able to answer people asking why we don't use it on the build daemons. > > It started with the removal of usrmerge from the archive (see #1088212), > > which broke the generation of the chroots on the build daemons using > > debootstrap. People on IRC just said that mmdebstrap should be used instead > > debootstrap. In the meantime an empty package has been reuploaded until > > debootstrap is fixed in stable. > > Okay, I see. I think some people are a bit too excited about mmdebstrap some > times. It has its place and so has debootstrap. I'd advocate for replacing > debootstrap by mmdebstrap everywhere. They are similar but still different. > For > example, the goal of mmdebstrap is to go away. In an ideal world (and Julian > is > experimenting with this right now), there would be an "apt bootstrap" command > which creates an initial chroot for you. In mmdebstrap I explore ways to do > that as mmdebstrap itself is just a wrapper around apt, passing it the right > options. In essence, you can replace mmdebstrap by doing this: > > mkdir -p "$2/etc/apt" "$2/var/cache" "$2/var/lib" > cat << END > "$2/apt.conf" > Apt::Architecture "$(dpkg --print-architecture)"; > Apt::Architectures "$(dpkg --print-architecture)"; > Dir "$(cd "$2" && pwd)"; > Dir::Etc::Trusted "$(eval "$(apt-config shell v Dir::Etc::Trusted/f)"; printf > "$v")"; > Dir::Etc::TrustedParts "$(eval "$(apt-config shell v > Dir::Etc::TrustedParts/d)"; printf "$v")"; > END > echo "deb http://deb.debian.org/debian/ $1 main" > "$2/etc/apt/sources.list" > APT_CONFIG="$2/apt.conf" apt-get update > APT_CONFIG="$2/apt.conf" apt-get --yes --download-only install '?essential' > for f in "$2"/var/cache/apt/archives/*.deb; do dpkg-deb --extract "$f" "$2"; > done > chroot "$2" sh -c "dpkg --install --force-depends > /var/cache/apt/archives/*.deb" > > This is not much. You could even replace debootstrap on the build daemons > with something like the above, just add one additional step installing > build-essential. Thanks for the detailed explanation, I never look at mmdebstrap internals, it's actually quite interesting to see how existing code, apt, is reused there. > > > How would your preferred solution look like? > > > > > > A proper solution would need to call "apt-get indextargets" on all > > > invocations > > > to figure out whether a https mirror was used, slowing down even non-https > > > runs. > > > > From what I understand the only protocol supported by apt that need > > ca-certificates is the https case, so it should be enough to just check > > the URL like debootstrap is doing. > > Unfortunately, in contrast to debootstrap, there is not "the URL". One of the > features of mmdebstrap over debootstrap is, that it supports multiple mirrors. > With debootstrap, you get only a single mirror url, so detecting https is > trivial. With mmdebstrap, not only do you have apt deb822 sources.list as > input, you also have hooks which can do arbitrary things. To cover these > cases, > you'd need to run: > > apt-get indextargets --format '$(URI)' > > This would probably happen at the very end, just before the apt cache is > cleaned up. Ok, the key point here is the hooks, I understand better now. That said, from what I understand the hooks are executed outside of the chroot/unshare, with the host apt, which has access to the CA certificates, so it should be enough to do determine that as the last step once all the hooks have been executed. But without knowing the mmdebstrap internals, I probably miss a point. > > > Are you using mmdebstrap from the terminal or from a script? In what > > > use-case > > > do you come across this issue? > > > > I used it from a script, as said above to replace the one used by the > > buildds. The script can be changed easily to pass > > --include=ca-certificates, but I believe that users trying to reproduce > > the build daemon setup might encounter the same issue. > > From the feedback I get from other users, mmdebstrap is usually used from a > script. This is also how I mostly am using it. This feature is not for users > of > mmdebstrap from a script because those can easily add > --include=ca-certificates > as you said. This is for users who manually run mmdebstrap from the command > line and who want to automatically create a chroot with the right set of > packages without having to type too much. > > My problem with this is indeed that implementing this properly means that > apt-get indextargets will be called by default. One of the main reasons I > found > that people recommend mmdebstrap is, that it is really fast. I also like this > feature and would like to keep it that way. Unfortunately, running apt-get > indextargets on my machine takes half a second which is considerable if > creating the whole chroot takes 9 seconds. That's more than 5% runtime > increase. Tests here shows that's more around 0.6% (0.08 sec over 13 sec), but i guess it depends on cpu/network/storage speed/ratio and many other factors. And at then end 1% plus 1% plus 1%... can end-up with something slow at at the end. > Also, if this is added by default, there needs to be a way to disable this > feature. That would probably be a new skip option like > --skip=customize/auto-mirror. > > Another possibility would be to create a hook that does the right thing and > then the user would have to add: > > --customize-hook=/usr/share/mmdebstrap/hooks/auto-https > > But that's much longer than --include=ca-certificates, so not a good option > for > interactive use. Yes, I fully agree that this is not really an issue for scripts, they can be adjusted, but it is more problematic for interactive users, either directly or probably through sbuild when adding extra chroots. > I probably have to sleep over this... > Unfortunately at this stage I also don't have any better idea to provide. Cheers Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://aurel32.net
signature.asc
Description: PGP signature
