Re: merged-/usr transition: debconf or not?

[Ansgar]

On Thu, 2021-11-18 at 23:01 -0500, The Wanderer wrote:
> * to date, package maintainers have not yet begun moving
> already-packaged files from / to /usr/ (specifically because doing so
> would break systems that have not yet been migrated to merged-/usr,
> and Debian has not yet declared that such systems are unsupported),

That claim is incorrect:

- Some packages have moved files from /(s)bin to /usr/(s)bin. (This
  sometimes requires a compat symlink.)

- More packages have moved files from /lib to /usr/lib. (This often
  doesn't require any extra care.)

> * after bookworm, package maintainers will start moving already-
> packaged files from / to /usr/, and

s/start/continue/

> * doing this will, in a non-negligible number of cases, trigger the
> bug to manifest on systems where that package is upgraded from a
> version where the move had not taken place to one where it has.

Why do you claim that?

Given packages already did such moves in the last years and you claim
this happens in a non-negligible number of cases, could you please
point to some examples where this already happens in practice?

Ansgar

Re: Consequences of the NEW queue's length [Was: Remove packages from NEW queue?]

[Gard Spreemann]

Simon Richter  writes:

> Hi,
>
> On 11/18/21 4:08 PM, Stephan Lachnit wrote:
>
>> I guess this raises the (maybe already answered) question if the
>> additional license QA from NEW is for the end-product (i.e. Debian
>> stable) or for the servers that run the Debian infrastructure, which
>> of course includes experimental.
>
> The latter.
>
> The license must allow Debian to redistribute the package. This is
> checked very thoroughly on the initial upload, and updates are
> expected to keep the same licensing. Whether that expectation still
> holds with upstreams who prefer vendor copies over using external
> packages is another matter, but in general these packages require more
> handholding anyway.

I struggle to see how this assumption is reasonable at all, even keeping
vendoring upstreams out of the picture. It is hardly uncommon for
non-giant projects to re-license themselves one or more times after the
initial Debian package has cleared the NEW queue. The larger our
archive, and the more time passes, the more packages we can expect to be
shipping whose d/copyright's relationship with reality was never checked
by the FTP Masters.


 -- Gard
 


signature.asc
Description: PGP signature

Re: Consequences of the NEW queue's length [Was: Remove packages from NEW queue?]

[Gard Spreemann]

Andrey Rahmatullin  writes:

> On Thu, Nov 18, 2021 at 04:08:23PM +0100, Stephan Lachnit wrote:
>> > > I don't know if that has been proposed before, but how about waiving
>> > > the NEW queue requirement for experimental packages as a start?
>> > > [...] Since packages in experimental will never land in any
>> > > official release, I think dropping the NEW queue requirement has no
>> > > negative impact.
>> > This makes no sense as NEW is mostly about checking licenses.
>> 
>> I think this is exactly why it makes sense. I think we can trust the
>> DDs to not make any large mistakes (e.g. putting steam in main).
> The existence of NEW means we currently don't, for completely new
> packages.
>
>> Since packages in experimental aren't supposed to be used by anyone else
>> but the DDs themselves, the "damage" of a potentially missing / wrong
>> license is minimal, considering that DDs are aware of the fact that the
>> packages aren't "official".
> The "damage" that's usually being discussed is Debian distributing
> something we can't, not users e.g. getting non-free software thinking it's
> free. Packages in NEW aren't even publicly accessible because of this,
> and discussions of switching to git-based source packages end with "we
> can't publish git history of random repos as we don't want to review
> and rewrite it".
>
>> However I find that view a bit weird. Any update can change the license
>> or add new files with different licenses, nothing is ever checked by the
>> ftp-masters (that would be insanity).
> Sure.

And in light of the above, assume:

 - Source package foo with a single binary package foo, entered Debian
   under thorough FTP Masters checking 20 years ago, has had a very
   active upstream and rapid development (including numerous rewrites
   and language changes) since then, but always just that single binary
   package.

 - Source package bar that ships a binary libbarN, entered Debian under
   thorough FTP Masters checking a few weeks ago after NEW queueing for
   months. Slowly developing upstream with few, very limited and
   organized, changes over time. Needs a SONAME bump. Or needs to add a
   new bar-utils binary, or something.

If the goal is to maximize our users' confidence that foo and bar can be
assumed DFSG-compliant, then it does not to me seem like the best use of
our (FTP Masters') limited resources to subject bar to another thorough
review (perhaps having it spend another 4 months in NEW?), while
trusting the maintainer of foo with no further scrutiny.

I'd go as far as to say that if we could not trust the maintainer of foo
the way we do, we could not reasonably make Debian work. And if we can
trust the maintainer of foo with a task where it's arguably easier to
make a mistake than it is for the maintainer of bar, and we cannot trust
the maintainer of bar in that safer case, then we are quite
inconsistent. I would argue that this inconsitency is problematic, both
because it leads to a misapplication of resources, because it may well
convey a false sense of safety onto our users, and because it may be
detrimental to attracting new volunteers.


 -- Gard


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Philip Hands]

Ansgar  writes:

>> * doing this will, in a non-negligible number of cases, trigger the
>> bug to manifest on systems where that package is upgraded from a
>> version where the move had not taken place to one where it has.
>
> Why do you claim that?
>
> Given packages already did such moves in the last years and you claim
> this happens in a non-negligible number of cases, could you please
> point to some examples where this already happens in practice?

My understanding is that in order to trigger this bug you need at least
to both move a file from one place to the other, and also to rename the
package that contains that file or move ownership to another package.

I suspect that you might also need to be unlucky with the order that
apt/dpkg decides to do the installation and, depending upon how far
apart the move and the rename happens, also unlucky with your choice of
from and to versions of the packages in question.

Given that these bugs are going to be utter bastards to reproduce, and
you can be sure that we'll have enough diversity in installed systems
that some people are going to manage to be sufficiently unlucky, it
would be nice to know the sort of damage we might expect.

It strikes me that we ought to be able to screen our own repos for
packages that could be able to tickle this bug. That would give us the
chance to look at what sorts of files we might realistically expect to
be clobbered, it should give some indication of how many packages we
should expect to be able to trigger this, and knowing this might suggest
plausible work-arounds.

Of course, that doesn't help with packages from third-party repos,
including our downstreams, but at present we seem to be discussing this
with very little hard data.

It occurs to me that one could lose quite a few files on the average
Debian install (if they were selected at random) without even noticing,
whereas a very few files would render systems unbootable, so knowing a
bit more about which files are realistically at risk would be very
helpful in understanding the severity of the problem.

If anyone's got good ideas about how to gather this information, I'm
very happy to help with the effort to do so.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[David Kalnischkies]

On Fri, Nov 19, 2021 at 10:06:13AM +0100, Ansgar wrote:
> Given packages already did such moves in the last years and you claim
> this happens in a non-negligible number of cases, could you please
> point to some examples where this already happens in practice?

You need a / → /usr¹ and a pkg-a → pkg-b move in the same release.
Also, you need to have (the new) pkg-b be unpacked before pkg-a.

An example³ would be coreutils/util-linux/… moving everything from /bin
to /usr/bin and in the same Debian release splitting out one (or more)
of their tools into their own package (as they usually do).
As those are essential they will Pre-Depend² on the split out package
which guarantees that pkg-b will be unpacked before pkg-a.

The result is that the split out tool will be gone from /usr-merged
systems – which at that point should be all systems.


Another example would be systemd files debhelper moved for some time
already while the package does a foo and foo-data split in the same
Debian release. Just need to "solve" the unpack order now, but I will
leave that as an exercise for the reader.



The move and reorganisation is both forbidden by the CTTE for Debian 12
in "Moratorium on moving files' logical locations into /usr" which even
describes this problem as one of the reasons for it, but hopes to have
it resolved by 13 (without mentioning how).

Are you suggesting that Debian will use 13 to move each and every
file in / to its /usr counter-path while forbidding that packages
including moves are reorganised before 14 ?

Good thing 'which' isn't in /bin I guess. (SCNR)

Disclaimer: I am as usual not arguing for switching into full speed
reverse mode. I would "just" prefer if we acknowledge that problems
exist we have to deal with. Its gonna be hard enough to actually resolve
them given all bridges have been burned down years ago by pretending its
not a problem that dpkg has no idea what is done behind its back to the
files its supposed to manage.

(The problem itself isn't unique⁴ to /usr-merge, so ideally it would be
 resolved regardless, but /usr-merge undoubtedly makes heavy use of it
 so in an ideal world those interested in it would not only acknowledge
 the problems but actually work together to resolve them.
 Sadly, that isn't the world we seem to be stuck in at all.)


Best regards

David Kalnischkies

¹ You could of course also move the other way around.
² You can achieve the same with other dependency types, it is just
  harder to trigger in simple testcases as apt & dpkg try to avoid
  the auto-deconfiguration of pkg-a if there is an easy way out.
³ For fun I wrote an apt testcase with coreutils splitting out ln⁴,
  that might be a bit unrealistic, but you get the idea (attached).
⁴ as, you know, /usr-merge being the last symlink we will ever need
#!/bin/sh
set -e

TESTDIR="$(readlink -f "$(dirname "$0")")"
. "$TESTDIR/framework"

setupenvironment
configarchitecture 'native'

#mkdir -p rootdir/bin
ln -s usr/bin rootdir/bin

touch ln

mkdir -p tree/bin
cp -a ln tree/bin
buildsimplenativepackage 'coreutils' 'native' '1' 'stable' '' '' '' '' 
'tree/bin'
rm -rf tree

buildsimplenativepackage 'coreutils' 'native' '2' 'unstable' 'Pre-Depends: 
unneeded-ln'

mkdir -p tree/usr/bin
cp -a ln tree/usr/bin
buildsimplenativepackage 'unneeded-ln' 'native' '2' 'unstable' 'Breaks: 
coreutils (<< 2)
Replaces: coreutils (<< 2)' '' '' '' 'tree/usr'
rm -rf tree

setupaptarchive

testfailure test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln
testsuccess apt install coreutils=1 -y
testsuccess test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln

testsuccess apt full-upgrade -y
testsuccess test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Michael Biebl]

On 19.11.21 11:58, Philip Hands wrote:

Ansgar  writes:


* doing this will, in a non-negligible number of cases, trigger the
bug to manifest on systems where that package is upgraded from a
version where the move had not taken place to one where it has.


Why do you claim that?

Given packages already did such moves in the last years and you claim
this happens in a non-negligible number of cases, could you please
point to some examples where this already happens in practice?


My understanding is that in order to trigger this bug you need at least
to both move a file from one place to the other, and also to rename the
package that contains that file or move ownership to another package.

I suspect that you might also need to be unlucky with the order that
apt/dpkg decides to do the installation and, depending upon how far
apart the move and the rename happens, also unlucky with your choice of
from and to versions of the packages in question.


Right. I think it would be immensely useful to have an actual reproducer 
so one could study the issue more closely or at least a bug report, 
which describes the issue in more detail, like the exact circumstances 
when this can happen.

So far this is merely theoretical, right?
Or do we have a documented instance of this happening?



Given that these bugs are going to be utter bastards to reproduce, and
you can be sure that we'll have enough diversity in installed systems
that some people are going to manage to be sufficiently unlucky, it
would be nice to know the sort of damage we might expect.

It strikes me that we ought to be able to screen our own repos for
packages that could be able to tickle this bug. That would give us the
chance to look at what sorts of files we might realistically expect to
be clobbered, it should give some indication of how many packages we
should expect to be able to trigger this, and knowing this might suggest
plausible work-arounds.

Of course, that doesn't help with packages from third-party repos,
including our downstreams, but at present we seem to be discussing this
with very little hard data.

It occurs to me that one could lose quite a few files on the average
Debian install (if they were selected at random) without even noticing,


Shouldn't debsums be able to detect such missing files (at least for the 
vast majority of packages which ship a md5sums file).


I run that semi-regularly on all of my systems, most of them are 
/usr-merged and I haven't noticed any missing files yet which I could trace



whereas a very few files would render systems unbootable, so knowing a
bit more about which files are realistically at risk would be very
helpful in understanding the severity of the problem.

If anyone's got good ideas about how to gather this information, I'm
very happy to help with the effort to do so.


I'd be more then happy to help here as well.

Regards,
Michael




OpenPGP_signature
Description: OpenPGP digital signature

Bug#1000208: ITP: pcmemtest -- stand-alone memory tester

[fzielcke]

Package: wnpp
Severity: wishlist
Owner: Felix Zielcke 
X-Debbugs-Cc: debian-devel@lists.debian.org

* Package name: pcmemtest
  Version : 1.5
  Upstream Author : Martin Whitaker
* URL : https://github.com/martinwhitaker/pcmemtest
* License : GPL-2
  Programming Lang: C, Assembly
  Description : stand-alone memory tester

 PCMemTest is a stand-alone memory tester for x86 and x86-64 architecture
 computers. It provides a more thorough memory check than that provided by BIOS
 memorytests.
 .
 PCMemTest can be loaded and run either directly by a PC BIOS (legacy or UEFI)
 or via an intermediate bootloader that supports the Linux 16-bit, 32-bit,
 64-bit, or EFI handover boot protocol. It should work on any Pentium class or
 later 32-bit or 64-bit CPU.
 .
 PCMemTest is a fork and rewrite of Memtest86+, which in turn was a fork of
 Memtest86.


I'm happy to maintain it inside a team or with co-maintainer(s).
I'm only DM so if someone has interest in sponsoring this, feel free to
contact me.

Re: merged-/usr transition: debconf or not?

[Marc Haber]

On Fri, 19 Nov 2021 11:58:39 +0100, Philip Hands 
wrote:
>Given that these bugs are going to be utter bastards to reproduce, and
>you can be sure that we'll have enough diversity in installed systems
>that some people are going to manage to be sufficiently unlucky, it
>would be nice to know the sort of damage we might expect.
>
>It strikes me that we ought to be able to screen our own repos for
>packages that could be able to tickle this bug. That would give us the
>chance to look at what sorts of files we might realistically expect to
>be clobbered, it should give some indication of how many packages we
>should expect to be able to trigger this, and knowing this might suggest
>plausible work-arounds.

This is one of the cases where I wish that Debian would be a more
centrally organized project. Red Hat or SuSE would just fix their
package management and go on with their business.

It's a pity that we have actually THINK about alternatives to that
trivial and obvious approach because we leave our core package
maintainers too much freedom to stall.

Grüße
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: merged-/usr transition: debconf or not?

[Sven Joachim]

On 2021-11-19 15:37 +0100, Michael Biebl wrote:

> On 19.11.21 11:58, Philip Hands wrote:
>> Ansgar  writes:
>>
 * doing this will, in a non-negligible number of cases, trigger the
 bug to manifest on systems where that package is upgraded from a
 version where the move had not taken place to one where it has.
>>>
>>> Why do you claim that?
>>>
>>> Given packages already did such moves in the last years and you claim
>>> this happens in a non-negligible number of cases, could you please
>>> point to some examples where this already happens in practice?
>> My understanding is that in order to trigger this bug you need at
>> least
>> to both move a file from one place to the other, and also to rename the
>> package that contains that file or move ownership to another package.
>> I suspect that you might also need to be unlucky with the order that
>> apt/dpkg decides to do the installation and, depending upon how far
>> apart the move and the rename happens, also unlucky with your choice of
>> from and to versions of the packages in question.
>
> Right. I think it would be immensely useful to have an actual
> reproducer so one could study the issue more closely or at least a bug
> report, which describes the issue in more detail, like the exact
> circumstances when this can happen.
> So far this is merely theoretical, right?
> Or do we have a documented instance of this happening?

#953562 seems to be such an instance.

Cheers,
   Sven

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Marco" == Marco d'Itri  writes:

Marco> On Nov 18, Zack Weinberg  wrote:
>> Are you seriously claiming that that phenomenon is not a
>> severity:critical bug?
Marco> I am seriously claming that whatever you are referring to, if
Marco> true, is such a contrived example that does not actually
Marco> happen in real life (or at least, it does not happen
Marco> frequently enough).

It was convincing enough to convince me, the TC and several other
participants in the discussion.

--Sam

Re: merged-/usr transition: debconf or not?

[Sam Hartman]

> "Michael" == Michael Biebl  writes:

Michael> Am 17.11.2021 um 19:57 schrieb Sam Hartman:
Michael> AIUI simply moving files from / to /usr within the same
Michael> package is not problematic.

Sorry, I was being overly simplistic.
I think your understanding is mostly correct.
You need to make sure that in the same release you do not both
move files from / to /usr

and then later move files between packages.

It's not just that you can't do it at the same time.
It's that it cannot happen within a given release cycle.

Your packaging team may be organized enough to make sure you don't do
that.
And I guess if you've got some plan to make sure you never trip the
problematic conditions it would be okay.

I don't think the project as a whole is taht organized.
So, for example, I think it would be problematic for general tools like
debhelper to move files, because we cannot guarantee that maintainers
will not later (or earlier) move those files between packages.

But we could you know fix dpkg:-)
I'm sure that will be painful at the political level, but personally I
think it needs doing.

Anyway, thanks for calling me out on being imprecise.  There's enough
FUD going around in this discussion I do not want to add to it.

Re: merged-/usr transition: debconf or not?

[Paul Gevers]

Hi

On 18-11-2021 22:44, Marco d'Itri wrote:

On Nov 18, Zack Weinberg  wrote:


Are you seriously claiming that that phenomenon is not a severity:critical bug?

I am seriously claming that whatever you are referring to, if true, is
such a contrived example that does not actually happen in real life
(or at least, it does not happen frequently enough).


I'm thinking of bug #972936 and bug #953562. Did I remember those 
wrongly as being an example of this?


Paul



OpenPGP_signature
Description: OpenPGP digital signature

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Richard Laager  writes:

> Is this particularly hard to fix, though?

> Can't we just do something like the following pseudo-code:

[...]

> (I've used /bin -> /usr/bin as an example, but the canonicalization must
> deal with the other merged paths too.)

> The second bit ensures that all new operations write canonicalized paths
> (e.g. /usr/bin/foo) into the database regardless of whether the .deb 
> shipped with /bin/foo or /usr/bin/foo. This ensures the database stays in
> sync with the filesystem moving forward.

> The first bit performs a one-time canonicalization of paths in the
> existing database. An on-disk flag ensures that this is run only once 
> (for performance reasons only, because it's already idempotent). This
> corrects the existing database-filesystem mismatch.

> The one-time canonicalization can be removed once non-usrmerged systems
> are no longer supported. The second bit needs to live (probably a lot) 
> longer, until it's no longer reasonable to install a .deb containing
> non-usrmerged paths (which might be old or from a third party).

> Am I missing something here?

I also don't understand why this isn't the correct solution.  It seems
obvious to me that we should simply teach dpkg about path aliases and have
it update both its internal state and its processing of new packages to
canonicalize paths so that it has an accurate representation of the world,
and then ensure dpkg is upgraded first.

Are we missing something?  If not, what is blocking this solution?  Is it
simply a matter of someone digging into dpkg's code sufficiently to put
these changes in the correct location?  Or is there some other issue?

It seems like a huge waste of resources to me to do archive-wide package
inspection to try to find patterns that might cause problems, and to ask
all maintainers to remember this rather obscure rule, when we could just
fix dpkg to make the problem go away.

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

On Thu, Nov 18, 2021, at 8:06 PM, Luca Boccassi wrote:
>> On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
>> Luca Bocassi wrote:
>> ...
>>> nobody has actually seen [the file disappearance bug]
>>> happen, to the best of my knowledge.
>>
>> I already explained why that doesn't prove the bug is a non-issue.
>> To the contrary; it means there is an enormous installed base of
>> systems where the bug is latent, waiting to cause problems under
>> conditions which we can reasonably expect to occur shortly after
>> the release of bookworm.
>
> Why would the release of bookworm make any difference?

Up until the release of bookworm, all Debian packages must be
constructed on the assumption that they _might_ be unpacked on a
system that has not yet been converted to merged /usr.  Particularly
for priority-required packages, this means that no one will be moving
files from /bin, /lib, etc to /usr in the bookworm cycle.

Post-bookworm, if nothing changes, that assumption will no longer be
in force, and people who maintain packages that install files into /
will want to simplify their packaging by installing everything in /usr
instead.  If they also want to change the binary packages that ships
some of those files at any point in the same release cycle -- kaboom.

zw

Re: merged-/usr transition: debconf or not?

[Zack Weinberg]

On Fri, Nov 19, 2021, at 3:23 PM, Zack Weinberg wrote:
> On Thu, Nov 18, 2021, at 8:06 PM, Luca Boccassi wrote:
>>> On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
>>> Luca Bocassi wrote:
>>> ...
 nobody has actually seen [the file disappearance bug]
 happen, to the best of my knowledge.
>>>
>>> I already explained why that doesn't prove the bug is a non-issue.
>>> To the contrary; it means there is an enormous installed base of
>>> systems where the bug is latent, waiting to cause problems under
>>> conditions which we can reasonably expect to occur shortly after
>>> the release of bookworm.
>>
>> Why would the release of bookworm make any difference?
>
> Up until the release of bookworm, all Debian packages must be
> constructed on the assumption that they _might_ be unpacked on a
> system that has not yet been converted to merged /usr.  Particularly
> for priority-required packages, this means that no one will be moving
> files from /bin, /lib, etc to /usr in the bookworm cycle.
>
> Post-bookworm, if nothing changes, that assumption will no longer be
> in force, and people who maintain packages that install files into /
> will want to simplify their packaging by installing everything in /usr
> instead.  If they also want to change the binary packages that ships
> some of those files at any point in the same release cycle -- kaboom.

After having caught up on the thread, I see that the conditions required
to trigger the bug are somewhat more complicated, but the point remains:
particularly for the packages where a lost file could render the system
unbootable, changes that would trigger the bug have been deferred until
post-bookworm, *and* we can reasonably expect the maintainers of those
packages to *want* to make changes with a high risk of triggering the bug.
I imagine the coreutils maintainers are looking forward to getting rid
of their list of programs that go in /bin, and the extra debian/rules
logic to go with it, for instance
(https://sources.debian.org/src/coreutils/8.32-4.1/debian/rules/#L16).

zw

Re: merged-/usr transition: debconf or not?

[Timo Röhling]

* Russ Allbery  [2021-11-19 11:46]:

I also don't understand why this isn't the correct solution.  It seems
obvious to me that we should simply teach dpkg about path aliases and have
it update both its internal state and its processing of new packages to
canonicalize paths so that it has an accurate representation of the world,
and then ensure dpkg is upgraded first.

+1


Are we missing something?  If not, what is blocking this solution?  Is it
simply a matter of someone digging into dpkg's code sufficiently to put
these changes in the correct location?  Or is there some other issue?

AFAIR, Guillem has not commented on the solution when it was brought
up on d-devel [1], so I'm not sure if he would accept patches, let
alone support anyone working on this. I'd be willing to assist him
and contribute code, but I'm not going to spend a good chunk of my
spare time just to see me work summarily rejected.

Cheers
Timo

[1] https://lists.debian.org/debian-devel/2021/08/msg00438.html


--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: merged-/usr transition: debconf or not?

[Simon Richter]

Hi,

On Fri, Nov 19, 2021 at 12:28:56PM -0700, Sam Hartman wrote:

> But we could you know fix dpkg:-)
> I'm sure that will be painful at the political level, but personally I
> think it needs doing.

I think the main blocker at the moment is that the dpkg change *also*
has the potential to break a lot of things if it isn't carefully
designed.

The problem space right now is huge:

 - for upgraded systems, the system could be pre-usrmerge, or the
   conversion could have been performed by any usrmerge version that
   ever existed (and the usrmerge package could have been deinstalled
   and reinstalled since, so the patches to the conffiles it performs
   could be inconsistently applied).
 - for upgraded systems, any version of usrmerge since the last stable
   release could be installed
 - for upgraded systems, unless the usrmerge package is discontinued,
   it could be upgraded at any point during the update.
 - for freshly installed systems, dpkg is first invoked on a
   non-usrmerged tree, and should convert the installation as soon as it
   becomes aware that conversion is desired (which might not be
   something we want to hardcode in dpkg, but possibly configure from a
   separate, Debian-specific package like base-files).
 - for freshly installed systems, initial unpack might be controlled by
   debootstrap from stable, which is usrmerge aware, so some of its
   workarounds may be active, and we need to properly capture this
   state, too.
 - freshly installed systems might be created with cdebootstrap or
   multistrap, and could be generated in --foreign mode
 - the dynamic loader is specified to be on the root filesystem, and that
   is part of the ABI standard and not under our control, so we cannot get
   a conflict-free dpkg database in bookworm.

>From dpkg's perspective, we now (kind of) change the policy for
directory-vs-symlink conflicts, which dpkg currently resolves in favour
of the directory. That was done in ancient times because it was somewhat
easy to do this in a package by accident. Not sure if that still
applies, but if it does, then we need to formulate the new policy so
that we don't catch a regression here (which is why my original idea to
just ship the symlinks in base-files is a bad idea).

>From the way dpkg and the Debian Policy are initially designed, it is
obvious to me that they started out as specification documents, and only
when these were hammered out, they were poured into code and rules, and
we've been operating from that stable base since, with two exceptions
where technical facts were created without updating Policy accordingly,
and which has both times been controversial.

My interpretation of the "political" situation is that we need this to be a
group effort, because no single person has the necessary time to do a
thorough enough job that they would feel comfortable signing off on it. I'd
be reluctant to do so even if I had three months of uninterrupted time --
that's the level of complexity I see here, and I suspect Guillem feels the
same.

So I believe the way forward will be writing a specification and giving it
enough eyeballs to identify problem cases. Finding the adversarial example
for the status quo was easy, since I had to find just one -- the goal now
is to get to a state where we don't find such an example easily anymore.

The specification should explain how the transition can be finished with
the bookworm release for all the different ways packages can be installed,
and it should describe the necessary code changes and new test cases to get
full coverage, and the document should be signed off by multiple people who
jointly take responsibility, to avoid placing all of that on a single
person -- because that's what's impeding progress right now, that no one
wants to be that person or even feels able to.

The description of the problem space I put above is likely incomplete and
overzealous at the same time (for example, I haven't checked how different
the previous usrmerge packages are, and which of them ever made it into a
stable release), so this is also just a starting point. I'll happily spend
time refining this or any other proposal, but I'm too time-constrained to
be the main driving force here -- this is not my day job, after all.

   Simon

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Simon Richter  writes:

> I think the main blocker at the moment is that the dpkg change *also*
> has the potential to break a lot of things if it isn't carefully
> designed.

I think you can simplify that problem space considerably because we know
exactly what symlinks we care about and we don't intend to support going
backwards (in other words, once you have merged /usr, it's not supported
to turn /bin, etc., back into regular directories again).

Therefore, dpkg can be configured with the list of the symlinks that
*must* exist for it to be willing to operate on the system.  Once it has
been put into merged-/usr mode (which includes updating its state database
to reflect the new canonical paths of all installed files), if those
symlinks do not exist or do not have the correct targets, the system state
is not safe and dpkg can (and probably should) abort and refuse to operate
on that system.

That modifies the problem of systems in various weird configurations from
"weird things may happen" to "dpkg will (safely) refuse to run" and we can
provide a tool to fix such a system by creating the correct symlinks.
(This tool may well just be the existing usrmerge tool.)  Getting into
that state will be hopefully rare if we structure the upgrade process so
that first one converts the system to merged-/usr if this has not already
been done, then upgrades dpkg, then tells dpkg to switch into merged-/usr
mode, and then safely upgrades the remaining packages.

> From dpkg's perspective, we now (kind of) change the policy for
> directory-vs-symlink conflicts, which dpkg currently resolves in favour
> of the directory.

I don't think we should be doing anything of the sort.  I think we should
introduce a new concept specifically for this case and have the remappings
of files for merged-/usr, once dpkg is in merged-/usr mode, take
precedence over any rules like that.  We do not want dpkg to be applying
some policy or heuristic about directory vs. symlink conflicts for
symlinks related to merged-/usr.  We know exactly what we want the rule
and behavior to be.  That sort of remapping should happen way before any
other dpkg analysis of the package or system state.

In other words, I argue that once dpkg is in merged-/usr mode, it should
treat a package with files in /bin exactly as if the files were in
/usr/bin, at an earlier stage than when it starts looking at the file
system.

We're not trying to create a generalized feature here.  Merged-/usr has a
very specific definition, with very specific and clear behavior that we
need from the package manager.  Trying to generalize this is going to make
a mess and make it much harder to reason about the behavior or test it.

-- 
Russ Allbery (r...@debian.org)  

Re: merged-/usr transition: debconf or not?

[Simon Richter]

Hi,

> > (I've used /bin -> /usr/bin as an example, but the canonicalization must
> > deal with the other merged paths too.)

/lib and /lib64 are a bit special, because they are part of the ELF ABI,
and any manipulation of these paths needs to be atomic.

Bootstrapping a new Debian system works by unpacking Essential packages
with ar+tar, i.e. not using dpkg. These packages will always be unpacked
to the file names listed inside the packages.

So, new systems will either need a package that ships the /lib ->
usr/lib symlink in the Essential set, or we need to ship the dynamic
linker in /lib, which would be a conflict during unpacking.

> > The second bit ensures that all new operations write canonicalized paths
> > (e.g. /usr/bin/foo) into the database regardless of whether the .deb 
> > shipped with /bin/foo or /usr/bin/foo. This ensures the database stays in
> > sync with the filesystem moving forward.

This loses the information about the original path, so this would be a
trapdoor operation that can not be undone without affected packages
going into reinstallation-required state (and we probably couldn't even
identify those).

Ideally, I'd rather not hardcode the list of top-level symlinks into
dpkg, because they might be architecture-specific, or change over time,
so there should be a mechanism to add and remove transformations.

> > The one-time canonicalization can be removed once non-usrmerged systems
> > are no longer supported. The second bit needs to live (probably a lot) 
> > longer, until it's no longer reasonable to install a .deb containing
> > non-usrmerged paths (which might be old or from a third party).

My expectation would be that this code will be here to stay, because it
will be required for bootstrap.

> Are we missing something?  If not, what is blocking this solution?  Is it
> simply a matter of someone digging into dpkg's code sufficiently to put
> these changes in the correct location?  Or is there some other issue?

If it were that simple, someone would have done that already. Dpkg has
an elaborate per-file tracking to recover from errors, and this change
introduces several new states and transitions, last but not least
because we also have filters and diversions as transformations on the
database -- but the usrmerge transformation is supposed to be orthogonal
to that.

> It seems like a huge waste of resources to me to do archive-wide package
> inspection to try to find patterns that might cause problems, and to ask
> all maintainers to remember this rather obscure rule, when we could just
> fix dpkg to make the problem go away.

That is the goal, yes, but this is a massive undertaking. We already
have a quick hack for usrmerge support in dpkg, which is the check that
allows moving a file from /bin to /usr/bin within the same package
without the file being deleted in the end (because deletions are
processed last for obvious reasons), and the obscure rule is precisely
the limitation of this hack.

I've stumbled across this hack while investigating if it was possible to
"just" fix dpkg. :P

   Simon

Re: merged-/usr transition: debconf or not?

[Russ Allbery]

Simon Richter  writes:

> Bootstrapping a new Debian system works by unpacking Essential packages
> with ar+tar, i.e. not using dpkg. These packages will always be unpacked
> to the file names listed inside the packages.

Well, bootstrapping a new Debian system involves running a tool that
bootstraps a new Debian system.  I think you're constraining the problem
too much.

It's a nice property that everything on the system comes straight from a
Debian package, but it's not a strict requirement, nor is it currently
generally the case (maintainer scripts do things, for instance).  Those
symlinks are very special from a dpkg perspective; dpkg needs to refuse to
mess with them even when upgrading a package that provides them, which
would mean some irritating special-casing I suspect.  It's not clear to me
that shipping them as tar members of a package is the right way to go, as
opposed to creating them separately as part of early system configuration.

> Dpkg has an elaborate per-file tracking to recover from errors, and this
> change introduces several new states and transitions,

Why?  I am not convinced this is true, apart from the obviously-needed
one-time conversion of the state database.

It involves a new filter, applied whenever a package is installed, that
changes all the paths in the package to their merged-/usr paths, something
that seems like it should be done before any other package processing.
Once that's done, why would any new states or transitions be required?

You could convince me that writing the filter is complicated because there
may not be one place in dpkg where you can currently put such path
rewriting since dpkg probably wasn't designed with that operation in mind.
But it's going to be harder to convince me that there are new states or
transitions; that smells like an over-complicated design.

> That is the goal, yes, but this is a massive undertaking.

I'm still not convinced of this.

> We already have a quick hack for usrmerge support in dpkg, which is the
> check that allows moving a file from /bin to /usr/bin within the same
> package without the file being deleted in the end (because deletions are
> processed last for obvious reasons), and the obscure rule is precisely
> the limitation of this hack.

This already sounds conceptually more complicated than just solving the
problem properly, with the caveat as mentioned above that writing the
necessary filter may be difficult.

-- 
Russ Allbery (r...@debian.org)  

Bug#1000240: ITP: golang-github-mhilton-openid -- openid OP endpoint implementation

[Mathias Gibbens]

Package: wnpp
Severity: wishlist
Owner: Mathias Gibbens 
X-Debbugs-CC: debian-devel@lists.debian.org, debian...@lists.debian.org

* Package name: golang-github-mhilton-openid
  Version : 0.0~git20181012.aeae87e-1
  Upstream Author : Martin Hilton
* URL : https://github.com/mhilton/openid
* License : Expat
  Programming Lang: Go
  Description : openid OP endpoint implementation

 openid provides a go implementation of the openid protocol,
 at the moment only the OP endpoint is implemented.

This is a dependency for packaging golang-github-canonical-candid (ITP
#998752), and will be team-maintained within the Debian Go Packaging
Team.


signature.asc
Description: This is a digitally signed message part

Bug#1000241: ITP: golang-gopkg-goose.v1 -- Go bindings for talking to OpenStack

[Mathias Gibbens]

Package: wnpp
Severity: wishlist
Owner: Mathias Gibbens 
X-Debbugs-CC: debian-devel@lists.debian.org, debian...@lists.debian.org

* Package name: golang-gopkg-goose.v1
  Version : 0.0~git20170406.3228e4f-1
  Upstream Author : Canonical Ltd
* URL : https://github.com/go-goose/goose
* License : LGPL-3.0
  Programming Lang: Go
  Description : Go bindings for talking to OpenStack

 Go OpenStack Exchange (goose) - Go bindings for talking to
 OpenStack.
 .
 This is a stable branch, which is compatible with the original
 goose source from Launchpad, except for the changed import paths.

This is a dependency for packaging golang-github-canonical-candid (ITP
#998752), and will be team-maintained within the Debian Go Packaging
Team.


signature.asc
Description: This is a digitally signed message part