Hi Simon,
On Fri, Jun 30, 2023 at 08:06:15PM +0900, Simon Richter wrote:
> I think "backports" are missing as a user story.
I fully agree. What a serious omission. As a first step, I have updated
DEP17 to indicate which mitigations happen to work when being
backported. For instance, changing Replaces to Conflicts is something
that happens to just work for backports. Diverting dpkg-divert less so.
So in effect, the most likely outcome is that backports become very
fragile, because they essentially have to undo all the moves performed
in unstable. What we can do with relative ease is detect the problems
after they have been introduced.
I note that the quality of backports already leaves wishes. For
instance, kodi-addons-dev takes over /usr/bin/dh_kodiaddon_depends in
bullseye-backports from kodi-addons-dev-common in bullseye without
conflicts nor replaces. Likewise, nfs-ganesha-ceph takes over
/usr/lib/ganesha/libganesha_rados_*.so in bullseye-backports from
nfs-ganesha in bullseye.
Is making backports more fragile a reasonable trade-off for moving
forward?
> Most packages should be harmless, and the Contents file for
> bullseye-backports doesn't have too much in any of the affected directories.
Yeah, the number of affected cases should be relatively low, but when
things go wrong, it's bad. Is that ok if we can automatically detect it
(after it happened)?
> but the list of packages installing files into /lib is longer and includes
> all the kernel backports, so I guess that is another potential source of
> problems.
Without having looked, I'd expect the majority of practically affected
files below /lib to be systemd units and udev rules. These should not be
moved in backports, and as long as debhelper is being used, that might
just work.
> There might be an easy solution here, I have not investigated this very
> deeply because it is a workday and 11 hours out of every day are already
> spoken for.
I don't think there is an easy solution, but maybe it happens to work by
chance 90% (number made up) of the time and we shall be able to detect
the remaining 10%.
> > Stating a goal has been quite difficult, but I think that most of the
> > participants agree that we want to eliminate the file move moratorium
> > without getting problems in return.
>
> I'd even widen that to "no more special handling needed in any place for the
> transition", with the moratorium being an example of that.
What other kind of special handling do you have in mind? I probably
agree.
> > When we get into mitigations, consensus is hard to come by. My
> > impression is that we have roughly two camps. One is in favour of major
> > changes to dpkg and the other is not.
>
> It's difficult to summarize the situation in one sentence, because neither
> group is really objecting to dpkg changes, so I'd put the fault line at
> whether the transition should be facilitated through dpkg or not.
It's not clear to me, whether you'd consider M3 (a minor and revertable
mitigation in dpkg) to be covered by this or not.
> > * Even if dpkg were fixed in trixie, trixie's libc6 would still be
> > unpacked using bookworm's dpkg. At least for this package, we cannot
> > rely on dpkg changes in trixie. Therefore we need workarounds or
> > spread the transition to forky. For other packages, even a
> > Pre-Depends on dpkg is not a guarantee that a changed dpkg will
> > perform the unpack.
> > * Changes to dpkg will not fix bootstrapping.
>
> The dpkg changes will fix bootstrapping, but we can't finish the transition
> until forky this way, because we need to be able to bootstrap with a libc6
> package that can be installed from bookworm.
Can you elaborate why? As far as I can see, debootstrap may perform the
initial unpack without help from dpkg. Then we invoke the unpakced dpkg
to configure essential packages. If dpkg plays any role in setting the
aliasing symbolic links, their creation will be late for running dpkg.
Maybe I'm missing something?
> We need to be careful here to not conflate the goal of the transition with
> the method for reaching it. We have consensus on the goal (basically,
> data.tar and filesystem matching is the definition of "done" I use for this
> transition).
I agree that we need to be careful about that conflation, because I
think you are conflating them. I disagree that the goal is having
data.tar and filesystem match up. You earlier argued that we are done
when special handling is no longer necessary and I see that as the goal.
Having all the files moved is one method to get there. We also seem to
have rough consensus that this is the preferre method.
> We do not have consensus on the technical implementation because there are
> people who believe the technical implementation proposed is not actually
> feasible. In my opinion, it is a 95% solution, which is very tempting but we
> need the remaining 5% as well. To a large extent, we are having this
> discussion because the usrmerge package left us with a 90% solution.
Yes.
> There is a third option: not changing the bootstrapping protocol, unpacking
> into an unmerged tree and merging as soon as possible, from one of the
> essential packages, so that the result of "debootstrap --merged-usr" and
> "debootstrap --no-merged-usr" is the same (which is also an excellent test).
This is true. In essence, this is what we do now as usrmerge happens to
be transitively-essential (when taking the first alternative). I kinda
dismissed this as a non-solution, because it's not actually reaching the
goal, but this approach still solves a significant chunk of practical
effects. Is that actually an acceptable final state where we have ~10
packages left that cannot move their files to canonical locations?
> > Any package that wants to rely on the new behavior requires a versioned
> > `Pre-Depends` on `dpkg`.
> > Doing this to `libc6` would introduce a dependency cycle.
>
> That is likely not a big problem, the only packages relying on the new
> behaviour are systemd-sysv and any packages that want to move files between
> packages and between paths at the same time, so basically we'd reduce the
> scope of the file move moratorium to a small set of core packages that have
> no reason to move files, and these would be moved in forky.
I think this directly yields:
Option #6a:
The /usr-merge transition can be unfinished in forky, but the number
of affected packages should be reduced to a minimum.
Option #6b:
We want the /usr-merge transition to be done by forky. In forky, we
may still need to remove transitional mechanisms used to facilitate
it.
It seemed obvious to me that #6b would be consensus, but it is not!
> > M3: Let `dpkg` use the filesystem as source of truth
> > ----------------------------------------------------
...
> Extending this check is probably a viable path, but because the check
> whether a file is known to dpkg requires a hash table lookup, we need a list
> of candidate file names, which requires dpkg to know the list of aliases.
> This has not been explored in depth yet, so there may be additional
> checkmarks in that table.
I subtly disagree. dpkg does not need to know the list of aliases.
Rather, it can look up the realpath of a file according to the
filesystem and thereby only use the implicit knowledge of relevant
aliases.
Thanks for this feedback in depth.
Helmut
