Bug#1036077: Further information

[Ben Zwick]

Please note that I have installed the skypeforlinux program and it 
segfaults when this problem occurs.

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Steve McIntyre]

Hey Johannes,

On Mon, May 15, 2023 at 06:48:04AM +0200, Johannes Schauer Marin Rodrigues 
wrote:
>Quoting Steve McIntyre (2023-05-15 02:54:02)
>> 
>> Pointing at gentoo or nixos as examples of projects that have decided
>> to break compatibility doesn't cut it, I'm afraid. They're well known
>> for changing fundamental things around Linux and (basically) not caring about
>> interoperability. That attitude is *not* Debian's.
>
>me and Luca have different ideas about how bootstrapping a Debian chroot should
>look like and I don't want to make an argument *for* changing PT_INTERP here as
>I think that keeping compatibility with others by following ABI is a good thing
>and because I think (and hope -- but Helmut is doing that analysis right now)
>that the debootstrap problem can be solved in a way I envision without changing
>PT_INTERP. But what I do not understand about the argument against Luca's
>proposal is:
>
>Obviously, with Luca's proposal, binaries from packages built with a different
>dynamic linker path in them would not work on distributions without merged-/usr
>symlinks. But if the property of stuff from Debian being able to run on
>non-Debian non-merged-/usr systems is an important one, then why was it okay to
>have merged-/usr as the default? Because with merged-/usr we already changed
>the interface contract for a lot of things because now binaries and libraries
>can also be found at other locations than on non-merged-/usr systems. A script
>with a /usr/bin/bash shebang built on and for Debian will not work on a system
>without the symlinks.

Despite the massive upheavals of merged-/usr in *other* ways, it's
actually a *minor* change as far as compatibility is concerned
here. The runtime linker (aka interpreter) is responsible for
resolving symbols and finding needed libraries. So long as *that* bit
works OK, then ~everything else should follow OK. This is how it's
possible to have things work across distros: binaries don't actually
care where the libraries live, or whether they came from rpm or deb
packaging, etc.

The issue at hand here is that the interpreter path is the most basic
thing that matters for this compatibility. Break this and *nothing*
can work.

>So did we not years ago decide, that the result of the "cross- and
>inter-project discussion" is, that everybody is going merged-/usr and that's
>why we need it too and that's why it is okay to build a system where binaries
>and scripts built for it just may not run on those other systems that do not do
>it?  With merged-/usr we already *did* "change fundamental things around" for
>reasons that are really not clear to me (but which i do not want to discuss
>here) and as a result did not "care about interoperability" (with those who do
>not also adopt it). In my own Debian work I so far only got extra work because
>of merged-/usr and I do not see the benefits (yet) and I was hoping that
>"changing fundamental things around Linux and (basically) not caring about
>interoperability" was *not* Debian's attitude but alas here we are.
>
>So have we not already burned the bridges to the non-merged-/usr world? Why was
>it okay back then to say "we can make this change because all other important
>players are doing merged-/usr so we can/have to as well". And now in the
>PT_INTERP discussion somehow we care again about those systems? I thought we
>already had the "cross- and inter-project discussion" about merged-/usr and
>because the result was "yes, go for it" we did it too. But if that is the case,
>why do we now care for a subset of the interoperability problems caused by
>merged-/usr for systems that don't have it?

This change is absolutely *not* needed to make merged-/usr work; if
anybody is claiming that it is, then they are not being 100% honest
with us. All the other distros doing merged-/usr have done it without
making this change, and it's also been working OK for us so far
without this change.

Breaking an agreed interface contract like this is axiomatically
*wrong* and will hurt us and the rest of the Linux ecosystem.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"You can't barbecue lettuce!" -- Ellie Crane

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 14:36, Steve McIntyre  wrote:
>
> Hey Johannes,
>
> On Mon, May 15, 2023 at 06:48:04AM +0200, Johannes Schauer Marin Rodrigues 
> wrote:
> >So did we not years ago decide, that the result of the "cross- and
> >inter-project discussion" is, that everybody is going merged-/usr and that's
> >why we need it too and that's why it is okay to build a system where binaries
> >and scripts built for it just may not run on those other systems that do not 
> >do
> >it?  With merged-/usr we already *did* "change fundamental things around" for
> >reasons that are really not clear to me (but which i do not want to discuss
> >here) and as a result did not "care about interoperability" (with those who 
> >do
> >not also adopt it). In my own Debian work I so far only got extra work 
> >because
> >of merged-/usr and I do not see the benefits (yet) and I was hoping that
> >"changing fundamental things around Linux and (basically) not caring about
> >interoperability" was *not* Debian's attitude but alas here we are.
> >
> >So have we not already burned the bridges to the non-merged-/usr world? Why 
> >was
> >it okay back then to say "we can make this change because all other important
> >players are doing merged-/usr so we can/have to as well". And now in the
> >PT_INTERP discussion somehow we care again about those systems? I thought we
> >already had the "cross- and inter-project discussion" about merged-/usr and
> >because the result was "yes, go for it" we did it too. But if that is the 
> >case,
> >why do we now care for a subset of the interoperability problems caused by
> >merged-/usr for systems that don't have it?
>
> This change is absolutely *not* needed to make merged-/usr work; if
> anybody is claiming that it is, then they are not being 100% honest
> with us. All the other distros doing merged-/usr have done it without
> making this change, and it's also been working OK for us so far
> without this change.

That is absolutely true, it is not mandatory. It is one possible
solution (of many) to a particular use case being sounded out, that's
all. I don't think it was mentioned by anybody as needed, if it was,
happy to clarify.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

Luca Boccassi  writes:
> On Mon, 15 May 2023 at 02:26, Russ Allbery  wrote:

>> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh,
>> major player in Linux distributions, and I'm not sure how much they
>> care about compatibility with anyone else.)

> This is a counter-example to confute the assertion that *everybody* does
> the same thing, which has been made multiple times. I'm not sure whether
> it's an experiment or not, I mean if you ask their users/developers I
> think they'd tell you it's very much production ready, but it is largely
> irrelevant: it exists, and that's the only reason why it was mentioned,
> as it shows that it is _possible_ to do that and be a working
> distribution. Doesn't imply it's automatically desirable, but that was
> not the intention.

Ah, okay, I'm happy to agree with that point: you can violate the ABI and
continue to be a working distribution.  (There are a lot of parts of the
ABI that if you violated them you would not have a working distribution,
but this is not one of them so far as I can tell.)

> Not quite: my argument is that binaries from these packages are not
> intended and not required to be ran on non-Debian systems, so there's no
> incompatibility introduced in the first place - everything still works
> where it is supposed to, exactly as it was before.

I think we're saying the same thing but quibbling over phrasing.  I'd put
that as saying that it's fine for the binaries of certain core Debian
packages to be incompatible with the ABI because they're not intended to
be used outside of Debian.  (In other words, I'm talking about
incompatibility as a concrete, testable property of a binary, and I think
you're talking about incompatibility as a more abstract concept of a
distribution.)

> No aggression intended whatsoever, sorry if it appeared that way. I am
> trying to understand what the rules are.

Well, the rule that I'd ideally set is don't break the ABI, even if it's
not obvious why breaking the ABI is a bad idea or you can't see any bad
consequences that could come from it, unless the reason for breaking the
ABI is absolutely central to the mission and purpose of Debian.

That said, it's not like we've never shipped a binary in Debian with a
different PT_INTERP.  (I vaguely remember that some programming language
uses PT_INTERP tricks for some sort of private binary scheme?  Objective
CAML or something?  I ran across it once years ago and can't remember the
details.  Also, IIRC klibc does some sort of PT_INTERP trick in some
situations that I don't remember the details of, although I don't think it
does that with general binaries.)  So I do see your point that you would
prefer the rule to be more pragmatic than that.

My counterargument is that this proposal seems to mostly be about avoiding
having to create a symlink at a critical point in the bootstrap process,
and while it's tricky to get the timing right (and therefore kind of
annoying), the resulting usable system has to have that symlink anyway (I
think there's no disagreement about that).  Not following the ABI for core
binaries seems like a scary change with unknown consquences to a bunch of
core packages to solve what looks like a relatively minor (if admittedly
annoying!) problem.

Note that the target of PT_INTERP on Debian is *already* a symlink, to
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 which was the multiarch path
that we actually want to use.  We're already ensuring compatibility with a
symlink and I think we should just keep doing that and not venture into
these waters.

>> The world does not divide neatly into supported and unsupported use
>> cases.  There are a lot of things I do to computers that I expect to
>> work in some situations but not in others.  That includes, say, having
>> a Debian chroot on some other OS and running binaries from that chroot
>> without going into the chroot.  Often that will absolutely not work.
>> Sometimes it will work, and it's convenient that it will work for some
>> obscure recovery situations or other weird one-off use cases.  I've
>> also copied files from working systems to broken systems running a
>> different distribution before, and there's a list of caveats as long as
>> my arm, but sometimes it's a quick fix for something.

> Vast, vast majority of binaries from existing packages will already
> not work out of the box in that use case though.

I'm not sure why you think this is true and it makes me wonder if maybe my
intuition about cross-distribution compatibility is wrong.  I would expect
to be able to copy, say, most (all?) binaries from coreutils from a Debian
system to some other distribution and run it (and that's exactly the sort
of binary that is useful in this kind of cross-distribution rescue case).
Is this not true today?  What breaks?

Note that we're not talking about complicated packages with lots of
runtime like, say, Emacs.  As I understand it your proposal wouldn't
change PT_INTERP for that binary anyway. 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
> People build things on Debian that are not Debian packages. People
> compile binaries on Debian, and expect them to work on any system that
> has sufficiently new libraries.

*raises hand*

Hello, I represent an example of those people. With my $work hat on,
I'm involved in maintaining a family of Debian derivatives (the Steam
Runtime) that is used to run Steam games on arbitrary distributions,
including but not limited to Debian. Some of these binaries are built
on a Debian derivative, but run on an arbitrary other distribution,
using a RPATH[1] to find their non-glibc dependencies.

At the moment those binaries are built in ancient environments (based
on Ubuntu 12.04 and Debian 8), but as newer versions of glibc become
ubiquitous, we'll want to start relying on newer versions of Debian
(which will still be very old versions *at the time*, but I'm sure that
by 2030 or 2040 we'll want to be using versions of Debian that, in 2023,
are not released yet). In this use-case, yes we do need to be using the
interoperable ELF interpreter path.

We were able to use (Ubuntu and) Debian for this *because* Debian is
a relatively "ordinary" distribution that tends to follow cross-distro
standards. The major counterexample is multiarch library paths, but
multiarch library paths have some compelling technical advantages, and
because there's no ambiguity about which architecture uses which directory,
they're actually better for interop in some ways than /usr/lib (which
is ambiguous, because the Red Hat family of distros expects to find 32-bit
libraries there, but the Arch family expects 64-bit libraries, and it's
not possible to construct a tree where both get what they want).

I've spent a lot of the last 5 years working on putting Steam games in
containers so that they'll work more reliably on all distros, including
Debian, with less reliance on weird library search paths (although we
still have to use binaries built on an ancient Debian derivative with a
non-trivial RPATH to bootstrap the container environment). Because of
constraints around recent GPUs needing current graphics drivers even
if running on an otherwise ancient library stack, Steam's container
runtime has to construct a hybrid environment where glibc is either the
one from the host or the one from the container runtime environment,
whichever is newer; and while doing that, we have experienced some
broken situations that were caused by distributions diverging from the
interoperable ELF interpreter. One concrete example is that Arch Linux
uses the interoperable ELF interpreter for *almost* all executables,
but uses a different ELF interpreter for executables from the glibc
source package, for whatever reason; we were able to work around this,
but for a while it caused Arch to fail to launch these containers where
Debian/Fedora/etc. could.

This is not something that any of the distributions involved is
going to formally support, so the overall system consists of various
unsupported-but-works-in-practice things happening; but there is
absolutely no chance of Valve/Steam shipping separate binaries for
each distro (there are too many distros for that, even if you don't
count source-based distros like Gentoo that have an almost unlimited
number of concrete instantiations as binaries). However loudly we
might insist that our small subset-of-a-subset is the one they should
be targeting, what we're going to get is binaries that work on "mostly
interoperable" distros. As a result, if we want games on Linux (and
anything else requiring binary interop) to continue to be a thing,
pragmatically we should aim to be one of those "mostly interoperable"
distros, and encourage our friends and/or rivals in other distros to
do the same. Otherwise, the most likely outcome is for developers to go
back to an attitude of "I'm not going to support Linux, too many random
differences" and releasing for Windows only, and we all lose out.

As a result of all this, I would strongly prefer our compiler to
default to hard-coding the same interoperable ELF interpreter that
glibc upstream and the various major distros have agreed on (for example
/lib64/ld-linux-x86-64.so.2 on amd64), and I would also prefer it if we
can use that interoperable path in all the binaries we ship, including
src:glibc and the rest of the transitively Essential set.

One way that I like to think about this sort of thing is that we have a
finite number of "weird Debianism" tokens available, and we should aim
to spend them on the things that give us the best cost/benefit ratio.
We've never considered changing the ELF interpreter to be one of those,
to the extent of having a /lib64 solely for its benefit (on amd64)
even though by policy we don't generally use lib64.

(Incidentally, Steam's container runtime is always a merged-/usr
environment, to provide an environment that maximizes the probability
of any given script or binary working correctly; but it 

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Simon McVittie]

On Mon, 15 May 2023 at 06:48:04 +0200, Johannes Schauer Marin Rodrigues wrote:
> Obviously, with Luca's proposal, binaries from packages built with a different
> dynamic linker path in them would not work on distributions without 
> merged-/usr
> symlinks. But if the property of stuff from Debian being able to run on
> non-Debian non-merged-/usr systems is an important one, then why was it okay 
> to
> have merged-/usr as the default? Because with merged-/usr we already changed
> the interface contract for a lot of things because now binaries and libraries
> can also be found at other locations than on non-merged-/usr systems. A script
> with a /usr/bin/bash shebang built on and for Debian will not work on a system
> without the symlinks.

pre-bookworm gcc writes /lib64/ld-linux-x86-64.so.2 into the ELF header
of amd64 binaries and I think post-bookworm gcc should continue to do so
even though that has never been the physical path to the ELF interpreter
on Debian (it's really /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2, or
historically a version-numbered path). People who are only testing on
Debian systems, even in pre-merged-/usr releases like Debian 9, could
already have been relying on /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
existing; and now that we're saying merged-/usr is mandatory,
people who are only testing on Debian >= 12 could also start
relying on /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 or
/usr/lib64/ld-linux-x86-64.so.2; but we arrange for both/all paths
to exist, and we advise developers that they should only rely on the
interoperable /lib64/ld-linux-x86-64.so.2, treating all other paths that
lead to the same binary as an implementation detail.

I don't think it's any different to say that both /usr/bin/sh and /bin/sh
exist and will work, but ask that everyone should continue to use /bin/sh
and treat /usr/bin/sh as an implementation detail.

The way I think about merged /usr is a variant of Postel's principle:
be conservative in what you require, but be liberal about what will work.
A merged-/usr system can run software that assumes merged-/usr, and can
also run software that doesn't (because of the compatibility symlinks);
a non-merged-/usr system can run software that doesn't assume merged-/usr,
but software that makes that assumption will sometimes fail to work on it.

Consistent with that, despite being in favour of merged-/usr myself,
I didn't switch my development laptop or my autopkgtest VM images
to the merged-/usr setup until it became effectively mandatory
during the bookworm cycle - because if a package was doing something
not-maximally-portable, I wanted my development laptop or my test VM to
be one of the places it would fail to work, so that I would notice and
report that bug.

Conversely, I *did* switch non-development computers (servers and family
laptops) to merged-/usr, because on those systems the important thing is
for software to work, even if in theory it "doesn't deserve" to work.

On post-bookworm, merged-/usr Debian systems, if we keep using #!/bin/sh
and #!/usr/bin/perl scripts, and avoiding #!/usr/bin/sh and #!/bin/perl
scripts, then I think that's a positive thing for cross-distro interop -
and using the interoperable path to the ELF interpreter (dynamic linker)
is completely consistent with that.

As far as I can see, post-bookworm Lintian will continue to warn about
the non-interoperable spellings like #!/usr/bin/sh and #!/bin/perl,
unless changed to special-case them.

Note that the paths that are canonical from the point of view
of cross-distro interoperability (like #!/bin/sh) are not always
the same as the paths that are canonical from the point of view of
realpath(). *At the moment*, they are usually canonical from dpkg's
point of view, but that won't be the case forever, and I think that's
fine. /lib64/ld-linux-x86-64.so.2 isn't considered to be canonical by
either realpath() or dpkg either (dpkg knows it's a symlink, even on
non-merged-/usr systems where /lib64 is a real directory), but it's
canonical for the x86_64-linux-gnu ABI and that's what we say is most
important.

> With merged-/usr we already *did* "change fundamental things around" for
> reasons that are really not clear to me (but which i do not want to discuss
> here) and as a result did not "care about interoperability" (with those who do
> not also adopt it).

Didn't we? With merged /usr, the "theoretically correct" #!/bin/sh
scripts that have always worked will continue to work, and additionally,
"incorrect" #!/usr/bin/sh scripts will *also* work. That sounds like
caring about interoperability to me!

The one piece of interop we lose with merged-/usr becoming mandatory is
that if a developer has only tested on Debian (and other merged-/usr distros
like Fedora), if they have used a less-interoperable spelling of the path
like #!/usr/bin/sh or #!/bin/perl, their testing will not highlight that
source of potential non-interop with others.

However, I don't think it's credible to claim 

Bug#1036125: ITP: virtme-ng -- Build and run specific kernels inside a virtualized snapshot of your live system

[Ricardo Ribalda Delgado]

Package: wnpp
Severity: wishlist
Owner: Ricardo Ribalda Delgado 
X-Debbugs-Cc: debian-devel@lists.debian.org, rica...@ribalda.com

* Package name: virtme-ng
  Version : 1.6
  Upstream Contact: Andrea Righi 
* URL : https://github.com/arighi/virtme-ng
* License : GPL-2.0
  Programming Lang: Python
  Description : Build and run specific kernels inside a virtualized
snapshot of your live system

virtme-ng is a tool that allows to setup a lab to experiment different
kernel versions in a safe virtualized environment, re-using the entire
filesystem of the running machine as a safe live-copy, without affecting
the real filesystem of the host.

virme-ng is a fork of virtme. It is more active than the legacy version. The
original author
has been contacted and they are ok with the fork and replacing the original
virtme from Debian
until he has time to maintain it again.

Once this package lands I will ask for the removal of the old virtme.

Re: RFP: virtme-ng -- Tool to build and run a kernel inside a virtualized snapshot of your live system

[Ricardo Ribalda Delgado]

Hi all

I think I have the first version of virtme-ng.

@Héctor Orón Martínez can you help reviewing and pushing
https://salsa.debian.org/ribalda/virtme-ng ?

Maybe you could also create salsa.debian.org/debian/virtme-ng


Thanks and regards!

On Wed, May 10, 2023 at 5:04 PM Emmanuel Arias  wrote:
>
>
>
>
> On Wed, May 10, 2023 at 11:41 AM Ricardo Ribalda Delgado 
>  wrote:
>>
>> Hi Emmanuel
>>
>>
>> On Wed, May 10, 2023 at 1:03 AM Emmanuel Arias  wrote:
>> >
>> > Upstream respond  without  objection. Is there any volunter  to this 
>> > transition?  If not i would happy to work on it.
>>
>> I can take care of it. I won't probably start until the weekend though
>
>
> Sounds good, thanks!
>
> Cheers,
>>
>>
>> Regards
>>
>> >
>> > Cheers
>> >
>> >
>> > El mar, 9 de may de 2023, 07:54, Emmanuel Arias  
>> > escribió:
>> >>
>> >> Oh, I did not note/check that virtme already exists in Debian.
>> >>
>> >> Anyway, I am interest in the package, so I will follow virtme/virme-ng 
>> >> project :-)
>> >>
>> >> El mar, 9 de may de 2023, 07:49, Ricardo Ribalda Delgado 
>> >>  escribió:
>> >>>
>> >>> On Tue, May 9, 2023 at 12:46 PM Andrea Righi 
>> >>>  wrote:
>> >>> >
>> >>> > On Tue, May 09, 2023 at 11:32:59AM +0200, Ricardo Ribalda Delgado 
>> >>> > wrote:
>> >>> > > Hi
>> >>> > >
>> >>> > >
>> >>> > > On Tue, May 9, 2023 at 11:28 AM Héctor Orón Martínez
>> >>> > >  wrote:
>> >>> > > >
>> >>> > > > Hello,
>> >>> > > >
>> >>> > > > El mar, 9 may 2023, 9:51, Andrea Righi 
>> >>> > > >  escribió:
>> >>> > > >>
>> >>> > > >> On Tue, May 09, 2023 at 09:30:54AM +0200, Héctor Orón Martínez 
>> >>> > > >> wrote:
>> >>> > > >> > Hello,
>> >>> > > >> >
>> >>> > > >> >   virtme already exists in Debian, what would be the benefit of 
>> >>> > > >> > virtme-ng
>> >>> > > >> > over virtme?
>> >>> > > >> >
>> >>> > > >> > https://salsa.debian.org/debian/virtme
>> >>> > > >> >
>> >>> > > >> > Regards
>> >>> > > >>
>> >>> > > >> The original virtme project is not maintained anymore
>> >>> > > >> (https://github.com/amluto/virtme), so we decided to fork the 
>> >>> > > >> project
>> >>> > > >> and continue the development / bug fixing in virtme-ng
>> >>> > > >> (https://github.com/arighi/virtme-ng).
>> >>> > > >>
>> >>> > > >> Some people are already using and contributing to virtme-ng and 
>> >>> > > >> there
>> >>> > > >> are plans to package it in SuSE.
>> >>> > > >>
>> >>> > > >> Honestly I don't know what would be the right procedure to 
>> >>> > > >> "obsolete"
>> >>> > > >> the old virtme package and replace it virtme-ng (if possible), but
>> >>> > > >> ideally it would be nice to do something like this. Any guidance 
>> >>> > > >> or
>> >>> > > >> suggestion is welcome.
>> >>> > > >
>> >>> > > >
>> >>> > > > I suggest we evaluate switching upstream from virtme to virtme-ng, 
>> >>> > > > on the Debian virtme package. I would not mind if you want to be 
>> >>> > > > added as uploader for the package.
>> >>> > > >
>> >>> > > > Ricardo has been working on virtme. What do you think?
>> >>> > >
>> >>> > > SGTM. Maybe we can send an email out of courtesy to the old virtme 
>> >>> > > author.
>> >>> >
>> >>> > All good to me as well! I already sent an email to the virtme author
>> >>> > (Andrew Lutomirski) to inform him that I was forking the project, but I
>> >>> > didn't get any response.
>> >>>
>> >>> That sounds good.
>> >>>
>> >>> >
>> >>> > Maybe I can try to ping him again and see if he's also happy about this
>> >>> > plan.
>> >>>
>> >>> May I suggest that when you ping them, tell them about the plans to
>> >>> replace virtme with virtme-ng on Debian and put  me on cc?
>> >>>
>> >>> Thanks!
>> >>>
>> >>>
>> >>> >
>> >>> > -Andrea

Re: RFP: virtme-ng -- Tool to build and run a kernel inside a virtualized snapshot of your live system

[Andrea Righi]

On Mon, May 15, 2023 at 10:45:15PM +0200, Ricardo Ribalda Delgado wrote:
> Hi all
> 
> I think I have the first version of virtme-ng.
> 
> @Héctor Orón Martínez can you help reviewing and pushing
> https://salsa.debian.org/ribalda/virtme-ng ?
> 
> Maybe you could also create salsa.debian.org/debian/virtme-ng
> 
> 
> Thanks and regards!

Thank you so much for this Ricardo, and thank you for picking 1.6, that
I just released a few hours ago. Perfect timing! :)

-Andrea

> 
> On Wed, May 10, 2023 at 5:04 PM Emmanuel Arias  wrote:
> >
> >
> >
> >
> > On Wed, May 10, 2023 at 11:41 AM Ricardo Ribalda Delgado 
> >  wrote:
> >>
> >> Hi Emmanuel
> >>
> >>
> >> On Wed, May 10, 2023 at 1:03 AM Emmanuel Arias  wrote:
> >> >
> >> > Upstream respond  without  objection. Is there any volunter  to this 
> >> > transition?  If not i would happy to work on it.
> >>
> >> I can take care of it. I won't probably start until the weekend though
> >
> >
> > Sounds good, thanks!
> >
> > Cheers,
> >>
> >>
> >> Regards
> >>
> >> >
> >> > Cheers
> >> >
> >> >
> >> > El mar, 9 de may de 2023, 07:54, Emmanuel Arias  
> >> > escribió:
> >> >>
> >> >> Oh, I did not note/check that virtme already exists in Debian.
> >> >>
> >> >> Anyway, I am interest in the package, so I will follow virtme/virme-ng 
> >> >> project :-)
> >> >>
> >> >> El mar, 9 de may de 2023, 07:49, Ricardo Ribalda Delgado 
> >> >>  escribió:
> >> >>>
> >> >>> On Tue, May 9, 2023 at 12:46 PM Andrea Righi 
> >> >>>  wrote:
> >> >>> >
> >> >>> > On Tue, May 09, 2023 at 11:32:59AM +0200, Ricardo Ribalda Delgado 
> >> >>> > wrote:
> >> >>> > > Hi
> >> >>> > >
> >> >>> > >
> >> >>> > > On Tue, May 9, 2023 at 11:28 AM Héctor Orón Martínez
> >> >>> > >  wrote:
> >> >>> > > >
> >> >>> > > > Hello,
> >> >>> > > >
> >> >>> > > > El mar, 9 may 2023, 9:51, Andrea Righi 
> >> >>> > > >  escribió:
> >> >>> > > >>
> >> >>> > > >> On Tue, May 09, 2023 at 09:30:54AM +0200, Héctor Orón Martínez 
> >> >>> > > >> wrote:
> >> >>> > > >> > Hello,
> >> >>> > > >> >
> >> >>> > > >> >   virtme already exists in Debian, what would be the benefit 
> >> >>> > > >> > of virtme-ng
> >> >>> > > >> > over virtme?
> >> >>> > > >> >
> >> >>> > > >> > https://salsa.debian.org/debian/virtme
> >> >>> > > >> >
> >> >>> > > >> > Regards
> >> >>> > > >>
> >> >>> > > >> The original virtme project is not maintained anymore
> >> >>> > > >> (https://github.com/amluto/virtme), so we decided to fork the 
> >> >>> > > >> project
> >> >>> > > >> and continue the development / bug fixing in virtme-ng
> >> >>> > > >> (https://github.com/arighi/virtme-ng).
> >> >>> > > >>
> >> >>> > > >> Some people are already using and contributing to virtme-ng and 
> >> >>> > > >> there
> >> >>> > > >> are plans to package it in SuSE.
> >> >>> > > >>
> >> >>> > > >> Honestly I don't know what would be the right procedure to 
> >> >>> > > >> "obsolete"
> >> >>> > > >> the old virtme package and replace it virtme-ng (if possible), 
> >> >>> > > >> but
> >> >>> > > >> ideally it would be nice to do something like this. Any 
> >> >>> > > >> guidance or
> >> >>> > > >> suggestion is welcome.
> >> >>> > > >
> >> >>> > > >
> >> >>> > > > I suggest we evaluate switching upstream from virtme to 
> >> >>> > > > virtme-ng, on the Debian virtme package. I would not mind if you 
> >> >>> > > > want to be added as uploader for the package.
> >> >>> > > >
> >> >>> > > > Ricardo has been working on virtme. What do you think?
> >> >>> > >
> >> >>> > > SGTM. Maybe we can send an email out of courtesy to the old virtme 
> >> >>> > > author.
> >> >>> >
> >> >>> > All good to me as well! I already sent an email to the virtme author
> >> >>> > (Andrew Lutomirski) to inform him that I was forking the project, 
> >> >>> > but I
> >> >>> > didn't get any response.
> >> >>>
> >> >>> That sounds good.
> >> >>>
> >> >>> >
> >> >>> > Maybe I can try to ping him again and see if he's also happy about 
> >> >>> > this
> >> >>> > plan.
> >> >>>
> >> >>> May I suggest that when you ping them, tell them about the plans to
> >> >>> replace virtme with virtme-ng on Debian and put  me on cc?
> >> >>>
> >> >>> Thanks!
> >> >>>
> >> >>>
> >> >>> >
> >> >>> > -Andrea

Standards compliance (Was: Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited))

[Jeroen Dekkers]

On Mon, 15 May 2023 02:42:27 +0200,
Luca Boccassi wrote:
>
> On Mon, 15 May 2023 at 01:14, Russ Allbery  wrote:
>
> > An obvious specific example of such a system would be one that didn't
> > merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other
> > path, but that's just one obvious example.  There may be others; the whole
> > point of an ABI is that you do not change things like this, not even if
> > you can't personally imagine why your change wouldn't be harmful.  There's
> > a whole process for changing an ABI that involves everyone else agreeing
> > as well, and unless one goes through that process, the ABI is what it is.
> > Debian not building ABI-compliant binaries would be highly surprising.
>
> That's self-evidently not true, as there are other distributions where
> that already happens, it's been already mentioned. Besides, we are not
> talking about sacred religious texts - the point is making things
> work. If they do, is it _really_ non-compliant/incompatible?

The point of a standard is definitely not that people ignore it and do things
that are in direct contradiction with the standard. The point of a standard is
that people agree on a specification and will also follow that specification.

Something working and something being compliant are different concepts. Software
can be compliant and not working and software can also be non-compliant and
working. Something that is in direct contradiction with a standard is definitely
non-compliant. Whether it works or not is not relevant for that.

Kind regards,

Jeroen Dekkers

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 16:18, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
> > On Mon, 15 May 2023 at 02:26, Russ Allbery  wrote:
>
> >> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh,
> >> major player in Linux distributions, and I'm not sure how much they
> >> care about compatibility with anyone else.)
>
> > This is a counter-example to confute the assertion that *everybody* does
> > the same thing, which has been made multiple times. I'm not sure whether
> > it's an experiment or not, I mean if you ask their users/developers I
> > think they'd tell you it's very much production ready, but it is largely
> > irrelevant: it exists, and that's the only reason why it was mentioned,
> > as it shows that it is _possible_ to do that and be a working
> > distribution. Doesn't imply it's automatically desirable, but that was
> > not the intention.
>
> Ah, okay, I'm happy to agree with that point: you can violate the ABI and
> continue to be a working distribution.  (There are a lot of parts of the
> ABI that if you violated them you would not have a working distribution,
> but this is not one of them so far as I can tell.)
>
> > Not quite: my argument is that binaries from these packages are not
> > intended and not required to be ran on non-Debian systems, so there's no
> > incompatibility introduced in the first place - everything still works
> > where it is supposed to, exactly as it was before.
>
> I think we're saying the same thing but quibbling over phrasing.  I'd put
> that as saying that it's fine for the binaries of certain core Debian
> packages to be incompatible with the ABI because they're not intended to
> be used outside of Debian.  (In other words, I'm talking about
> incompatibility as a concrete, testable property of a binary, and I think
> you're talking about incompatibility as a more abstract concept of a
> distribution.)
>
> > No aggression intended whatsoever, sorry if it appeared that way. I am
> > trying to understand what the rules are.
>
> Well, the rule that I'd ideally set is don't break the ABI, even if it's
> not obvious why breaking the ABI is a bad idea or you can't see any bad
> consequences that could come from it, unless the reason for breaking the
> ABI is absolutely central to the mission and purpose of Debian.
>
> That said, it's not like we've never shipped a binary in Debian with a
> different PT_INTERP.  (I vaguely remember that some programming language
> uses PT_INTERP tricks for some sort of private binary scheme?  Objective
> CAML or something?  I ran across it once years ago and can't remember the
> details.  Also, IIRC klibc does some sort of PT_INTERP trick in some
> situations that I don't remember the details of, although I don't think it
> does that with general binaries.)  So I do see your point that you would
> prefer the rule to be more pragmatic than that.
>
> My counterargument is that this proposal seems to mostly be about avoiding
> having to create a symlink at a critical point in the bootstrap process,
> and while it's tricky to get the timing right (and therefore kind of
> annoying), the resulting usable system has to have that symlink anyway (I
> think there's no disagreement about that).  Not following the ABI for core
> binaries seems like a scary change with unknown consquences to a bunch of
> core packages to solve what looks like a relatively minor (if admittedly
> annoying!) problem.
>
> Note that the target of PT_INTERP on Debian is *already* a symlink, to
> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 which was the multiarch path
> that we actually want to use.  We're already ensuring compatibility with a
> symlink and I think we should just keep doing that and not venture into
> these waters.

It's not even a proposal, it's a discussion to see "what would
actually break if X happened". That's why I'm asking for concrete use
cases, rather than theoretical points. Also, I am interested in what
the rules are. For example, glibc compatibility is just as important
if not more, why does that follow a different standard? If anything,
adding a missing symlink is trivial and a single command. Adding a
missing symbol to a shared object... not quite. There is an endless
list of downstream-only debianisms that break cross-compatibility in
just the same way (if not worse), and yet nobody cares about those.
Why?

> >> The world does not divide neatly into supported and unsupported use
> >> cases.  There are a lot of things I do to computers that I expect to
> >> work in some situations but not in others.  That includes, say, having
> >> a Debian chroot on some other OS and running binaries from that chroot
> >> without going into the chroot.  Often that will absolutely not work.
> >> Sometimes it will work, and it's convenient that it will work for some
> >> obscure recovery situations or other weird one-off use cases.  I've
> >> also copied files from working systems to broken systems running a
> >> different distribution before, an

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 18:54, Simon McVittie  wrote:
>
> On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
> > People build things on Debian that are not Debian packages. People
> > compile binaries on Debian, and expect them to work on any system that
> > has sufficiently new libraries.
>
> *raises hand*
>
> Hello, I represent an example of those people. With my $work hat on,
> I'm involved in maintaining a family of Debian derivatives (the Steam
> Runtime) that is used to run Steam games on arbitrary distributions,
> including but not limited to Debian. Some of these binaries are built
> on a Debian derivative, but run on an arbitrary other distribution,
> using a RPATH[1] to find their non-glibc dependencies.
>
> At the moment those binaries are built in ancient environments (based
> on Ubuntu 12.04 and Debian 8), but as newer versions of glibc become
> ubiquitous, we'll want to start relying on newer versions of Debian
> (which will still be very old versions *at the time*, but I'm sure that
> by 2030 or 2040 we'll want to be using versions of Debian that, in 2023,
> are not released yet). In this use-case, yes we do need to be using the
> interoperable ELF interpreter path.
>
> We were able to use (Ubuntu and) Debian for this *because* Debian is
> a relatively "ordinary" distribution that tends to follow cross-distro
> standards. The major counterexample is multiarch library paths, but
> multiarch library paths have some compelling technical advantages, and
> because there's no ambiguity about which architecture uses which directory,
> they're actually better for interop in some ways than /usr/lib (which
> is ambiguous, because the Red Hat family of distros expects to find 32-bit
> libraries there, but the Arch family expects 64-bit libraries, and it's
> not possible to construct a tree where both get what they want).
>
> I've spent a lot of the last 5 years working on putting Steam games in
> containers so that they'll work more reliably on all distros, including
> Debian, with less reliance on weird library search paths (although we
> still have to use binaries built on an ancient Debian derivative with a
> non-trivial RPATH to bootstrap the container environment). Because of
> constraints around recent GPUs needing current graphics drivers even
> if running on an otherwise ancient library stack, Steam's container
> runtime has to construct a hybrid environment where glibc is either the
> one from the host or the one from the container runtime environment,
> whichever is newer; and while doing that, we have experienced some
> broken situations that were caused by distributions diverging from the
> interoperable ELF interpreter. One concrete example is that Arch Linux
> uses the interoperable ELF interpreter for *almost* all executables,
> but uses a different ELF interpreter for executables from the glibc
> source package, for whatever reason; we were able to work around this,
> but for a while it caused Arch to fail to launch these containers where
> Debian/Fedora/etc. could.

This sounds like a very interesting use case, and the first real one
mentioned, which is great to see - but I do not fully follow yet, from
what you are saying it seems to me that what you need is for your
binaries to use the usual pt_interp, that bit is clear. But why does
it matter if /usr/bin/ls on the host uses a different one? It's your
executables that you ship as part of that runtime that are the entry
points that need the usual loader path for your chroot-on-steroids,
no? The loader would still be reachable as it always was in this
theoretical exercise. I am probably missing something in how this
works in details.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Russ Allbery]

I'm dropping the TC bug from this thread, since I don't think it has
anything to do with that discussion at this point.  I probably should also
change the Subject line, but I'm keeping it to make it easier for the
people who want to tune out this thread, since I very much doubt they want
to tune back in at this point.

Luca Boccassi  writes:
> On Mon, 15 May 2023 at 16:18, Russ Allbery  wrote:

>> Note that we're not talking about complicated packages with lots of
>> runtime like, say, Emacs.  As I understand it your proposal wouldn't
>> change PT_INTERP for that binary anyway.  We're presumably talking
>> about the kind of binaries that you need to bootstrap a minimal system,
>> so packages like coreutils or bash.  And I would indeed expect those
>> binaries to be generally portable, as long as the same critical shared
>> libraries are available on other systems (in this case, PCRE2 and
>> ncurses).

> Is that really the case? Let's test that hypothesis:

I think you may have not read my whole message before replying, and also
please assume that I know really basic facts about glibc compatibility and
am not referring to that.

I said "of a similar vintage" (farther down in my message) because of
course we all know that binaries built against newer versions of glibc
don't run on systems with older versions of glibc (and likewise for shared
libraries in general and thus libselinux), and you tested a Debian
unstable package on an Ubuntu system from 2020.  This says nothing
interesting and has nothing to do with my point.

> Whoops. Does this make coreutils rc-buggy now? ;-)

You are the only person who is talking about RC bugs.  The bar here is not
"prove to me that this is RC buggy," the bar is "you have to prove to a
bunch of Debian core maintainers that they should break the ABI in their
packages" (not to mention the additional small but permanent build
complexity).  Demanding they prove to you that it's a bad idea is not how
this works.

The point of standards like an ABI is that a bunch of entirely unrelated
people who never talk to each other and never look at each other's
software are allowed to rely on them and assume no one else will break
them.  This is how free software scales; without invariants that everyone
can rely on without having to explain how they're relying on them, it is
much more difficult to get an ecosystem to work together.  We don't just
break things because they don't seem important; the space of people who
may be relying on this standard is unknowable, which is the entire point.
Opening those boxes is really expensive (in time, planning, communication,
debugging, and yes, compatibility) and we should only do it when it
really, really matters.

> It did look like a veto to me. More importantly, isn't relying on
> passersby to spot alleged harmful changes dangerous, especially for
> undocumented, uncodified and untested use cases, like unspecified and
> vague cross-compatibility requirements?

I'm honestly boggled.  This is a thread on debian-devel, which is
literally how we do architecture vetting in this project.

I absolutely do not think that we can write down everything of importance
in designing a distribution so that we don't have to have conversations
with other people in the project who have deep expertise when considering
a significant architectural change like changing the PT_INTERP path of
core binaries.

>> I mostly jumped in because it felt like you and Steve were just yelling
>> at each other and I thought I might be able to explain some of where he
>> was coming from in a way that may make more sense.

> I don't believe I've done any yelling here.

I'm using yelling pretty broadly and probably rather inaccurately here.
Maybe a better way of putting it is that Steve was yelling and you didn't
appear to be listening or understanding why he was yelling and were
responding in ways that were guaranteed to make him yell more.

You *are* coming across as kind of contemptuous of other people's
arguments, but then it's entirely possible that I am as well, so I'm
trying to ignore it.

> What if "setting the parking brake" is not enough, as the wheels are
> already off and rolling downhill, as shown above, because while
> everybody was looking at the parking brakes lever somebody ran off with
> the bolts that kept the wheels attached? Why is it worth worrying about
> compatibility with something that is already not compatible, and it's
> not expected to be compatible for almost all other aspects?

You can certainly decide to try to make the argument that ABI
compatibility between Linux distributions is a lost cause and we should
give up and not care about it any more.  That is a valid architectural
argument.  (To be clear, I don't think this is the argument you've been
making up to this point, which is about a very limited ABI break in
specific packages to achieve a specific purpose.)  I don't agree with that
argument, which doubtless doesn't come as a surprise, and your
justi

What *is* the amd64 ABI

[Sam Hartman]

We've all been talking about the x86_64 ABI, and I'm realizing I'm never
read it.
Are we talking about
https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf


--Sam

Re: What *is* the amd64 ABI

[Russ Allbery]

Sam Hartman  writes:

> We've all been talking about the x86_64 ABI, and I'm realizing I'm never
> read it.
> Are we talking about
> https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf

That's the one that I was looking at, at least.  Specifically (for the
purposes of the current discussion) page 84.  I'm not sure if there's a
newer version, since that one is labeled draft.

-- 
Russ Allbery (r...@debian.org)  

Re: What *is* the amd64 ABI

[Michael Hudson-Doyle]

On Tue, 16 May 2023, 15:42 Russ Allbery,  wrote:

> Sam Hartman  writes:
>
> > We've all been talking about the x86_64 ABI, and I'm realizing I'm never
> > read it.
> > Are we talking about
> > https://refspecs.linuxbase.org
> 

/elf/x86_64-abi-0.99.pdf
> 
>
> That's the one that I was looking at, at least.  Specifically (for the
> purposes of the current discussion) page 84.  I'm not sure if there's a
> newer version, since that one is labeled draft.
>

I think the current version lives at
https://gitlab.com/x86-psABIs/x86-64-ABI

Cheers,
mwh

-- 
> Russ Allbery (r...@debian.org)  
>
>

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Bastian Blank]

Hi Steve

On Mon, May 15, 2023 at 02:01:15AM +0100, Steve McIntyre wrote:
> Russ has described copying *binaries* out of packages and running them
> elsewhere. I've done that too, from time to time. This is one of the
> things made possible by the ABI contract being followed.

And nothing in that requires that the interpreter matches.  Because
glibc is clever enough to allow running the interpreter standalone.  So
in any way you can just call:

/lib64/ld-linux-x86-64.so.2 /bin/ls

Bastian

-- 
Oh, that sound of male ego.  You travel halfway across the galaxy and
it's still the same song.
-- Eve McHuron, "Mudd's Women", stardate 1330.1