Hi Guillem,
Thanks for taking the time to process this crazy idea!
On Wed, Mar 30, 2016 at 01:29:15AM +0200, Guillem Jover wrote:
> > b) Packages that do not "set -u" (nounset), can now prepend $DPKG_ROOT
> > to any file they operate on. With old versions $DPKG_ROOT will be
> > unset and with change a) $DPKG_ROOT will be empty. Thus this change
> > is backwards-compatible.
>
> Right, and these can always be set like ": ${DPKG_ROOT:=}". We could
> even recommend this as part of some doc/howto/spec for the initial
> deployments, before packages can assume a recent enough dpkg.
I like the recommendation, but I note that only about 30 packages employ
"set -u" (i.e. about 0.1%).
> > c) dpkg should gain a new force flag. I call it --force-remote-configure
> > for now. It is supposed to force dpkg into running maintainer scripts
> > without chroot even when the package in question did not declare that
> > its maintainer scripts support this mode of operation. Note that we
> > currently have no way to express whether a package supports running
> > maintainer scripts without chroot. The flag is being added by
> > 0002-add-force-remote-scripts.patch and the behavior is implemented
> > by 0003-inhibit-chroot-when-force-remote-scripts.patch. Packages can
> > only reasonably support this mode after implementing b).
>
> I don't quite like the name, as remote to me implies on some other
> machine. Perhaps foreign, host or extern(al), although some of these
> are a bit overloaded terms already.
Of course, I am not attached to the name. I just needed some string that
remotely made sense. What about "chrootless-configure" to make it
crystal clear? It should go hand in hand with d) if possible. What is
your preference?
> Also I'm not sure this makes sense as a force option (instead of its own
> proper option), as it is a behavior change that we want to always be safe
> to use, in contrast to a force option that just forces the behavior.
> Having a force option that would not chroot some times is a bit
> strange. We might still want to have both though, and there was a
> related bug report for that (#614126).
For the same reasons that you bring forward against making this a force
option, I think it should be a force option: No package in the archive
is currently known to be safe to configure from outside the chroot. So
doing that is inherently unsafe. We generally use force options to do
unsafe things. I see this force option as a development tool and not as
switch being used regularly. I think it is similar to --force-depends:
Proceed even though the package declared that it doesn't support the
mode of operation. The declaration is the absence of a support field as
in d). Does this make sense to you?
Possibly another switch is needed to enable this feature at all? I
thought we could just make it default (for supported packages), but
maybe that has downsides as well.
> > d) Once a) is accepted and b) starts getting implemented, we need to
> > think about a way for packages to tell that they support "remote
> > scripts". One way to do so would be to add a header "Remote-Scripts:
> > yes" to the binary package stanza. Packages thus marked would be
> > required to honour DPKG_ROOT in all maintainer scripts. This flag
> > makes no provisions yet on what programs can be assumed to be
> > installed outside the chroot that is operated on.
>
> Yes, ideally only packages marked as such would get a chroot-less
> environmemt when requested with the new option, because expecting the
> user to know when a package supports this mode and on what specific
> version is a terrible interface IMO, as it requires the user to
> analyze the .debs before unpacking them.
This also seems in support of the force option to override.
> One tiny concern is that if eventually we end up with the whole archive
> (haha? :) supporting this, then the field seems a bit of bloat.
I considered this as well, but thought of it as the lesser evil:
* I do not think that the whole archive will support DPKG_ROOT in a
reasonable time frame. The major benefit comes to essential and then
motivation to process further packages drops.
* We can still require DPKG_ROOT via policy and drop the field after
all oldstable packages support it.
> The other thing, that ISTR you brought up on IRC are triggers. I'm not
> sure how we'd handle those either. :/
I thought about this some more and the only sensible approach for
triggers I could come up with is handling them on their own: The
triggered package must declare support for this new mode or its trigger
processing will proceed in the old way (unless forced). The major
downside here is that you cannot tell beforehand whether you can
configure a package without chroot, but I don't see that changing
without breaking the flexibility of triggers.
> > e) Once a) is accepted and b) starts getting implemented, we need to
> > think about what programs maintainer scripts can assume to be
> > available outside the chroot. Some ways to handle that:
> > * Packages may only assume "common unix functionality". Such a set
> > would have to be defined somehow and roughly equates what
> > debootstrap requires.
>
> Defining this in terms of strict POSIX compliance (or a subset of the
> utilities defined within) would be the easiest/best I think.
I brought this forward as an option, but it really is one that I don't
like: Crucially, ldconfig is a glibc-ism and not POSIX. Thus the
libc-bin trigger cannot be converted and the utility of the whole
approach is fairly limited. In particular, it no longer addresses the
motivating use cases (debootstrap and multistrap).
> > * Packages may only assume essential packages to be available.
>
> That would resitrict the external system to be a Debian system. Which
> might be fine I guess but is more limited than what debootstrap requires.
> It has the problem that even then you need to specify essential of what
> Debian version?
Valid concern. I have no good idea on how to address it. Maybe this
approach just is a bad idea.
> > * A new set of headers Maint-{Depends,Conflicts,...} is added to
> > request tools to be installed. These new relations would be
> > checked outside the chroot (if any).
> >
> > A full Debian release or two needs to pass before such headers can
> > be used in the archive. This also poses the problem that a user
> > can remove packages required for removing other packages and thus
> > revoking the ability to remove certain packages. It is not clear
> > how the absence of Maint-Depends is supposed to be handled. It is
> > not clear whether dpkg needs to lock the dpkg database outside the
> > chroot.
I do have an answer to the absence of Maint-Depends now: Also add
Runtime-Depends. Then Depends would simply beam both Maint-Depends and
Runtime-Depends like Build-Depends means both Build-Depends-Arch and
Build-Depends-Indep. I note that even without the rest of the changes,
the splitting of Depends would make deity (or at least Don) a little
happier.
> As I mentioned at the time I don't think this is a workable solution.
> These are the problems I see right away with it:
At the same time, I think it is the only way to really improve the
problems with debootstrap and multistrap.
> * dpkg would need to load two status databases in memory, which would
> seriously complicate the code.
Yes.
> * dpkg would need to lock the external database, which means multiple
> chroots could not be used concurrently.
dpkg-checkbuilddeps does not lock the external database. Why would dpkg
have to?
> * As you mention, the dependencies might disappear under dpkg's feet,
> while it is not running in this mode.
dpkg-checkbuilddeps does not guarantee that Build-Dependencies do not
disappear during build (which can take much longer than package
installation), so why would dpkg require this?
What is the point in making different assumptions on dpkg and on
dpkg-checkbuilddeps? Both construct something external to the current
installation.
> * It ties the dependency graph from one installation into another,
> which can seriously complicate upgrades and similar.
In my picture, those dependencies on the outer systems are nothing dpkg
can do anything about. It just verifies them. If they aren't met, there
are basically two options:
* Fail.
* Proceed configuring with chroot as usual.
Also from the point of creating small Debian installations, splitting
Depends into pieces would be preferable. rpm already does that:
Requires(post): foo
This would allow us to strip an essential installation of packages only
required for configuring packages.
You see, I have a strong preference for allowing arbitrary packages to
be required from the outer installation.
> > With the above patches, I verified that I can install a package with a
> > maintainer script into a chroot that has an empty database (in
> > particular no essential package is unpacked in the chroot). The
> > maintainer script is run without chroot and has DPKG_ROOT set up
> > properly.
>
> Thanks for the patches! I think 2 and 3 need to be merged together,
> but see my comments on the name and if this deserves to be a force
> option or something else. And they need man page updates.
I can certainly squash those patches after you agreed on a name.
Documentation is missing indeed. Put on my stack.
> > bash's preinst will become a problem with this scheme. It is a binary
> > due to earlier breakage when it was a script, but the "remote scripts"
> > approach cannot work with binaries, because we cannot know whether the
> > cpu supports executing such binaries. I don't have a plan for
> > bash.preinst.
>
> I don't think the cpu support is a problem because I'd not expect
> actual remote execution to take place? But the linked libraries
> certainly are. We might perhaps need to read the maint scripts and
> only allow this mode if they are actual interpreted scripts (i.e. they
> have a shebang of any type). But that can be a safe guard added later.
bash's preinst really is a problem to my primary use case. I want to use
this feature to create e.g. an armhf sid chroot on an amd64 machine.
Currently, people tend to use multistrap + qemu-user-static for this,
but some architectures don't support qemu. So really, bash is a problem
as I'd want to execute a bash:armhf preinst on amd64.
I don't think we need a safe guard though, as bash simply wouldn't
declare support for this feature until some solution is found.
> Yeah doing a) by itself now seems pretty innocuous to me as well, so
> I'd be fine with including that, but the rest seems still pretty
> undefined.
Good. Likely a) also needs some form of documentation then. Do you see a
good place for that documentation? In the long term it would live in the
policy, but only after agreeing on the support declaration d).
I'd hope that we could also merge the other patches soonish, because
* It seems that their functionality is non-controversial. The pieces
that are related to dependencies and other aspects, but changing the
value of DPKG_ROOT and dropping the chroot call appear to be safe.
* It enables exploration. Only using it practically will tell us where
the real problems are.
* It is safe as the new functionality is never enabled by accident.
Helmut
