Re: DEP17 /usr-move: debootstrap set uploaded

[Luca Boccassi]

On Thu, 6 Jun 2024 at 10:02, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi Helmut,
>
> Quoting Helmut Grohne (2024-06-06 09:28:52)
> > I have just uploaded
> >  * base-files
> >  * bash
> >  * dash
> >  * glibc
> >  * util-linux
> > to unstable. These were the last remaining packages shipping aliased
> > files inside the package set relevant to debootstrap.
>
> thank you (and freexian for funding you) so much for finally reaching this
> milestone!! Thank you also for doing all your work with incredible diligence
> and attention to detail. This is really outstanding and what I consider to be 
> a
> model for how changes in Debian should be performed.
>
> Your upload timing seems to have been great: the buildds seem to already have
> gone through most of what you uploaded.
>
> I cannot wait for the next dinstall in ~5 hours to test what you uploaded with
> the mmdebstrap test suite (which is also testing debootstrap).
>
> Too bad that it is just these days that snapshot.d.o is moving to the new
> infrastructure (a big thanks to the team who is working on that) which means
> that this crucial transition in unstable will not be part of
> snapshot.debian.org. There are no timestamps recorded for June yet.

+1 thank you Helmut, excellent work!

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Luca Boccassi]

On Thu, 6 Jun 2024 at 09:07, Gioele Barabucci  wrote:
>
> Hi,
>
> setting LC_ALL=C.UTF-8 in d/rules is a common way to fix many
> reproducibility problems. It is also, in general, a more sane way to
> build packages, in comparison to using whatever locale settings happen
> to be set during a build. However, sprinkling a variant of `export
> LC_ALL=C.UTF-8` in every d/rules is error-prone and a waste of
> maintainers' time.
>
> Would it be possible to set in stone that packages are supposed to
> always be built in an environment where LC_ALL=C.UTF-8, or, in other
> words, that builders must set LC_ALL=C.UTF-8?
>
> In which document should this rule be stated? Policy?

This makes sense to me, seems similar enough to SOURCE_DATE_EPOCH

File descriptor hard limit is now bumped to the kernel max

[Luca Boccassi]

Hi,

PSA: as of systemd/256~rc3-3 the open file descriptors hard limit is
bumped early at boot from 1048576 to the max value that the kernel
allows, which on amd64 is currently 1073741816.

(I meant to send this last week but it fell off the wagon and
languished in the draft folder, sorry!)

This allows modern applications to use as many file descriptors as they
want, since they are an extremely cheap resource nowadays, and it's
more important than ever given that, for example, process tracking is
switching over to PID FDs for security reasons. Please note that the
soft limit still is 1024, as that's what legacy syscalls like select()
can handle.

The last time this was tried some packages were still not ready, so it
was patched out to let them be fixed. Enough time has passed now, and
it's time to let any unknown leftover just break and be fixed. In all
known cases, the buggy pattern was to manually iterate over the hard
limit and close every FD one by one, which is completely unnecessary
since kernel 5.9 (bullseye/oldstable) since the close_range() syscall
is available, that can do it in one fell swoop. Any packages still
doing the iteration manually need to switch to close_range, which is
very simple to use and documented at:

https://man7.org/linux/man-pages/man2/close_range.2.html

Please enjoy your extra file descriptors responsibly.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: Re: About i386 support

[Luca Boccassi]

> I've tried reproducing the daemon-reload bug report, unless I missed
> something
> obvious, daemon-reload works on my T2300, the TM Efficeon, and the
> pre-SSE2
> Pentium 3 (mobile) that I have. I could try running it on an original
> Pentium, but I doubt that debian will run on it at all, even when
> ignoring the fact that the thing also only has 96M of ram, which is
> to small to load a ramdisk and debian only targets i686. So the bug
> might only apply to a very specific processor, unless there is a
> patch
> in the debian package.

There are no relevant patches in the Debian package. This is exactly
the problem with supporting old and obsolete architectures, that are
very difficult to find in the wild: things break in weird and
incomprehensible ways, and nobody is able to fix them. This is one of
the main jobs of porters: if you can't triage and fix this issue, then
it's likely you won't be able to triage and fix other architecture-
specific issues either, as this is very very likely a hidden compiler
toolchain issue. The effort required to have a release architecture
officially supported in Debian goes way beyond "I have an old machine
under the desk and can build some trivial packages", I am afraid.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: Re: About i386 support

[Luca Boccassi]

On Fri, 14 Jun 2024 at 11:54,  wrote:
>
> Then it's not a problem in the first place. If you can't reproduce a bug with 
> a reasonable effort, then it is unconfirmed and you can stop worrying about 
> it. A bug that can't be reproduced, effectively doesn't exist.
>
> That's not a reason to stop supporting an entire architecture. That's a 
> troubleshooting decision that you would make on any architecture.

Of course it is a reason to drop an architecture. Not the mere
existence of an individual bug of course, but the fact that there is
nobody who is able or willing to successfully triage and debug and fix
such toolchain issues. As per RT's requirements, a release
architecture requires multiple porters as one of the basic
requirements to be considered, and the fact that nobody can fix issues
like these means there are effectively 0 porters. Once again,
maintaining a release architecture does not just mean having a machine
under your desk and building trivial packages, it means maintaining a
functioning toolchain, which involves triaging and fixing complex
bugs.

Re: File descriptor hard limit is now bumped to the kernel max

[Luca Boccassi]

On Fri, 14 Jun 2024 at 13:21, Mourad De Clerck  wrote:
>
> > PSA: as of systemd/256~rc3-3 the open file descriptors hard limit is
> > bumped early at boot from 1048576 to the max value that the kernel
> > allows, which on amd64 is currently 1073741816.
>
> Hi,
>
> It seems some proprietary software (the JetBrains IDEs) has some
> problems with this change.
>
> See for instance: https://youtrack.jetbrains.com/issue/IJPL-156522
>
> While I wait for them to fix this on their end, what's the best way to
> revert this to the original behaviour on my machine?
>
> I would think:
>
> echo "fs.nr_open = 1048576" > /etc/sysctl.d/99-max-fds.conf
>
> … would do the trick, but "ulimit -Hn" reports 524288.
>
> Something to do with DefaultLimitNOFILE=1024:524288 maybe? But
> overriding that didn't work.

For user instances the link you shared has a workaround, it has to do
with PAM limits, that should work.
Please keep the pressure on the upstream project to fix that bug as
well. Thanks.

Re: ifupdown maintenance

[Luca Boccassi]

On Tue, 9 Jul 2024 at 14:44, Simon Richter  wrote:
>
> Hi,
>
> On 7/9/24 17:57, Matthias Urlichs wrote:
>
> >> Agreed: either it's drop-in compatible or we may as well switch the
> >> default to NM and/or systemd-networkd.
>
> > Well, here's a heretical thought: why don't we do that anyway, at least
> > for new installations?
>
> Both are overly complex for a static-IP-only server installation, where
> there is no requirement for an unprivileged user to modify the network
> configuration, or any background process at all, really. At least in
> expert mode I would expect a way to generate a static,
> unmanaged-by-anything configuration.

As per smcv's point, if you need to manually write a static
configuration then you can also install your favourite tool to use it.
This is not the default case - the default case is "I have ethernet
and/or wifi and I want DHCP v4+v6 on anything that can talk to a
router kkthxbye"

> What would be needed for new installations is d-i support, mainly, and
> the difficulty there is saving the configuration.
>
> I believe NM does not have a fixed configuration format, but only a dbus
> API. Our best bet there would be a firstboot unit, but I have no idea
> whether accessing NM from a unit attached to firstboot is safe or leads
> to a dependency loop[1].
>
> I'm not sure if systemd-networkd is much happier long-term if we write
> its configuration from a shell script, we'd need to get some commitment
> from upstream that this is an interface, not an implementation detail,
> at least.

I'm not sure what this means, you can write networkd configuration
files from wherever you like, the tool doesn't care, it wouldn't even
be able to tell the difference anyway, just drop them in the
appropriate location in /etc/ and off you go.

Re: ifupdown maintenance

[Luca Boccassi]

On Tue, 9 Jul 2024 at 18:02, Simon McVittie  wrote:
>
> On Tue, 09 Jul 2024 at 16:21:16 +0200, Ansgar 🙀 wrote:
> > On Tue, 2024-07-09 at 22:44 +0900, Simon Richter wrote:
> > > I believe NM does not have a fixed configuration format, but only a dbus
> > > API.
> >
> > It's perfectly fine to edit configuration files for NM manually, see
> > man:nm-settings-keyfile(5).
>
> ... and debian-installer already knows how to write out these files,
> and has done so for more than a decade if I'm reading correctly. This is
> not a recent innovation, and anyone who has installed our default desktop
> environment in the last few years - especially if they used wifi during
> installation - has been relying on this code path.
>
>  is responsible
> for converting the network configuration that was used temporarily in
> the d-i environment into a NM configuration file if NM is installed,
> or an ifupdown configuration otherwise. Writing the equivalents into
> /etc/systemd/network for systemd-networkd would presumably not be rocket
> science, it's a simple .ini-like syntax similar to the one NM uses.

It is indeed quite simple:

https://salsa.debian.org/installer-team/netcfg/-/merge_requests/11

And I think you are perfectly right: for the simple, default use case,
we want something that is not downstream-only and dependent on a
single overworked maintainer, and that is modern enough to know that
Netlink exists and cannot be ignored. It's not 1998 anymore, even the
simplest systems need to take it into account, and there are
distro-agnostic solutions that everyone can rely on with minimal
integration efforts.

I think it's fine to add support for netplan, in fact it should be
merged first in the installer and my MR builds on top of it, and there
are many cases where it is a good idea to use it, but it should not be
the default, for the same reason as above - only Ubuntu really uses
it, and it does happen that Canonical's priorities shift.

The default logic in the above MR, absent specific choices, is: if
netplan is installed it will be used (needs to be preseeded or
installed manually, so it works for the cloud case), if
network-manager is installed it will be used (perfect for the
desktop/GUI case), if the interface is wired and it's on Linux and
systemd is init networkd will be used (perfect for minimal
installations/servers), else it goes back to the current ifupdown.

The main obvious advantage of networkd is that it is already installed
and will always be already installed in the default case, it's just
disabled, so the extra cost is one ini file and a couple of symlinks,
so the footprint of the minimal working install goes down. When
ifupdown loses prio: important then the footprint will actually
decrease.

Those who want complex cases can manually pick netplan. Those who want
to relive the thrills of the late 90s can pick ifupdown and write all
the crappy scripts they like, and spend their own time wondering why
eth0 became eth1 or viceversa and everything stopped working. The
default is the smallest and most sensible option. Everybody wins.

Re: what about Netplan?

[Luca Boccassi]

On Thu, 11 Jul 2024 at 12:12, Lukas Märdian  wrote:
>
> On 11.07.24 11:13, Marco d'Itri wrote:
> > On Jul 11, Philip Hands  wrote:
> >
> >> I've only seen netplan mentioned in passing in this thread so far.
> > Because I believe that Netplan is the answer to a question that nobody
> > asked here.
> >
> > It has all the disadvantages of switching to a new configuration format,
> > but then it limits you to the features that it actually implements from
> > each backend and you have an indirection layer that must be used when
> > interacting with the backend daemon.
>
> Yes, it brings a new configuration format. So do systemd-networkd and 
> NetworkManager.
>
> Netplan's feature set is pretty comprehensive and certainly enough for a 
> default installation,
> see its reference: https://netplan.readthedocs.io/en/stable/netplan-yaml/
>
> But it's not an indirection layer that MUST be used. Should the features of 
> Netplan not be enough
> for an advanced usecase, everybody is free to write configuration for the 
> underlying backend directly.
>
> Netplan will not interfere with that and go out of the way. Just don't write 
> Netplan configuration
> for a specific interface (or all of them) that you want to handle differently.

I appreciate the work you've done on Netplan and I am quite sure your
D-I MR should be merged, it looks ready to me. It's good to have this
support in the installer.

However, I do not think it should be the default. First of all, only
Ubuntu uses it, nobody else - as Simon says, we don't want the
defaults to be super-special things that nobody else uses. And then
again, not your fault in the slightest, but Canonical is known for
shifting its priorities every now and then - perfectly normal for a
for-profit company, but a bit worrying when one wants to pull all eggs
in that basket. And also for the minimal case, an additional layer of
indirection and set of dependencies, when there's an alternative that
requires none of that, is not the ideal solution.

So I am quite sure that yes netplan should be supported in the
installer and your MR should be merged, but the new default for the
default, simple, wired case should be networkd, and for the
GUi/desktop case the default should continue to be NetworkManager.

Re: what about Netplan?

[Luca Boccassi]

On Sun, 14 Jul 2024 at 11:41, Andrey Rakhmatullin  wrote:
>
> On Sat, Jul 13, 2024 at 09:44:03PM +0200, Lukas Märdian wrote:
> > > However, I do not think it should be the default. First of all, only
> > > Ubuntu uses it, nobody else - as Simon says, we don't want the
> > > defaults to be super-special things that nobody else uses. And then
> >
> > Actually, I think this is an agrument FOR Netplan, not against it. Netplan 
> > is being used
> > by millions of users for 7+ years. Plenty of usecases have been tried and 
> > documented. It's
> > clearly not a "super-special thing that nodbody uses".
> >
> > Whereas I'm not aware of a major Linux distro using systemd-networkd 
> > directly, Debian would be
> > singeling out itself. I see some of networkd's strengths with advanced 
> > users who want to dig deep
> > and have full control at minimal resource usage (e.g. Arch Linux). Also 
> > with lightweight container
> > usecases, where network config only needs minimal manipulation after 
> > deployment (if at all).

It largely depends on the configuration and flavours, in some cases
networkd is used for headless installs, in some other cases
network-manager is used, like in Fedora. Up until some time ago SUSE
used to use wicked, but they also switched to network-manager by
default somewhat recently, and wicked is deprecated. But nobody apart
from Ubuntu uses anything but network-manager on GUI installations at
this point. Debian is actually following the rest of the ecosystem on
this for once, and that is a good thing. I am quite convinced we
should stop being outliers, there are way more interesting things to
do with our limited time. It's fine to have other less popular options
available, even in the installer, but the default is something
different.

> > The RedHat ecosystem is all-in on NetworkManager. Debian and Ubuntu have 
> > (natually) been very close
> > to each other (e.g. package management) and together with its derivatives 
> > create the Debian ecosystem.
>
> Then it looks like a chance for netplan to go the way of upstart?

Or MIR or Unity or...

It's perfectly normal and expected for companies to follow their own
strategy and do what's best to pursue it. When things are aligned with
the rest of the ecosystem, it's not a problem. But when it goes in
opposite directions, then it's quite a different story. Recently there
was also the LXD case, where after many years a CLA and a license
change were introduced last year by Canonical, creating de-facto a
split, with the community choosing to fork to Incus - I do not know
the background details, as I am not involved in the slightest, and I'm
sure there must have been some reason for those decisions, but from an
outsider's perspective I'm afraid the optics were not quite good. What
guarantees do we have that what happened to LXD won't happen to
netplan.io at some point in the future?

Networking is not static, it constantly changes in the kernel,
sometimes in dramatic and incompatible ways. A widely used, well
maintained stack with large amounts of contributors is fundamental for
the default choice, because we have to keep up, as the rest of the
world will not sit and wait for us.

Here's some stats from 'git shortlog --after="2021-12-31" -sn --all'.
In the last ~2.5 years, in netplan.io's github repo, there are only 2
contributors with more than 100 commits, and 2 with more than 10, and
2 of them are Canonical employees:

   569  Lukas Märdian
   310  Danilo Egea Gondolfo
39  Simon Chopin
38  Danilo Egêa Gondolfo
11  Robert Krátký

Same stat, for the same period, for systemd:

  6650  Yu Watanabe
  5415  Lennart Poettering
  2884  Luca Boccassi
  2772  Zbigniew Jędrzejewski-Szmek
  2437  Daan De Meyer
  1793  Frantisek Sumsal
  1364  Mike Yuan
   483  Jan Janssen
   400  David Tardon
   245  Franck Bui
   215  dependabot[bot]
   211  Antonio Alvarez Feijoo
   165  Ronan Pigott
   152  Dan Streetman
   146  Ludwig Nussel
   126  hulkoba
   119  Nick Rosbrook
   114  Dmitry V. Levin
   107  Sam Leonard
   102  Evgeny Vereshchagin
78  msizanoen
74  Richard Maw
73  Adrian Vovk
72  Maanya Goenka
63  Michal Sekletár
60  Cristian Rodríguez
49  Michal Koutný
40  Jan Macku
40  Krzesimir Nowak
37  Mariano Giménez
37  Michael Biebl
37  Topi Miettinen
36  наб
35  Susant Sahani
33  Peter Morrow
32  Benjamin Franzke
32  Christian Brauner
32  Richard Phibel
31  Christian Göttsche
29  Anita Zhang
29  Khem Raj
26  James Hilliard
25  Abderrahim Kitouni
23  Arthur Zamarin
23  Florian Schmaus
22  Bastien Nocera
22  Daniel P. Berrangé
22  James Coglan
20  Arseny Maslennikov
20  G

Re: what about Netplan?

[Luca Boccassi]

On Sun, 14 Jul 2024 at 21:26, Philip Hands  wrote:
>
> Luca Boccassi  writes:
>
> > Here's some stats from 'git shortlog --after="2021-12-31" -sn --all'.
> > In the last ~2.5 years, in netplan.io's github repo, there are only 2
> > contributors with more than 100 commits, and 2 with more than 10, and
> > 2 of them are Canonical employees:
> >
> >569  Lukas Märdian
> >310  Danilo Egea Gondolfo
> > 39  Simon Chopin
> > 38  Danilo Egêa Gondolfo
> > 11  Robert Krátký
> >
> > Same stat, for the same period, for systemd:
>
> If you're going to make such a comparison, wouldn't it make sense to
> limit it to the bits that might have some sort of relation to networkd?
>
> I suspect this is a bit nearer to a fair comparison:
>
>   git shortlog --after="2021-12-31" -sn --all src/network/ network/
>771  Yu Watanabe
>107  Luca Boccassi
> 86  Zbigniew Jędrzejewski-Szmek
> 42  Lennart Poettering
> 32  Mike Yuan
> 24  Susant Sahani
> 18  Frantisek Sumsal
> 16  Daan De Meyer
> 13  Ronan Pigott
> 10  Jan Janssen
> 10  Michael Biebl
>
> which looks like its in the same ballpark, and presumably still includes
> quite a lot of stuff that would fall outside netplan's scope, so one
> could perhaps argue should be whittled down further.

Not at all: the way we structure systemd's repository, most of the
code that ends up in the network (or any other) binary are actually in
the shared areas. So if you really wanted to catch only the code that
ends up in the specific binaries:

$ git shortlog --after="2021-12-31" -sn --all src/network/ network/
src/libsystemd/ src/basic/ src/shared/ src/fundamental/

  2858  Yu Watanabe
  2195  Lennart Poettering
   882  Zbigniew Jędrzejewski-Szmek
   772  Luca Boccassi
   724  Daan De Meyer
   409  Mike Yuan
   196  Frantisek Sumsal
   128  Dan Streetman
87  David Tardon
76  Jan Janssen
64  Franck Bui
36  Nick Rosbrook
34  Cristian Rodríguez
31  Dmitry V. Levin
30  Adrian Vovk
25  Antonio Alvarez Feijoo
24  Susant Sahani
21  Topi Miettinen
20  msizanoen
19  Arseny Maslennikov
19  Ronan Pigott
17  Khem Raj
16  Ludwig Nussel
15  Maanya Goenka
13  Heinrich Schuchardt
12  Curtis Klein
12  Sam Leonard
11  Alberto Planas
11  David Rheinsberg
11  Florian Schmaus
11  rhellstrom
11  наб
10  Rafaël Kooi
10  jcg

and even that is not an accurate or good metric: the actual sources
that end up in a binary are not the only thing that matters in a
project. Documentation, manpages, unit tests, integration tests, CI,
build system, release tools, dev tools and more are just as important
to the health of a project, and are just as necessary (if not more) to
maintain it, and need active contributors. So the project-wide metric
is really the best metric to use for these kind of comparisons, for
these reasons, and for what Simon said too, of course.

Re: what about Netplan?

[Luca Boccassi]

On Sun, 14 Jul 2024 at 19:21, Simon McVittie  wrote:
>
> On Sun, 14 Jul 2024 at 17:09:48 +0100, Steve McIntyre wrote:
> > Luca Boccassi wrote:
> > >Networking is not static, it constantly changes in the kernel,
> > >sometimes in dramatic and incompatible ways.
> >
> > Sorry, but no. Networking clearly is *not* changing that fast, in
> > software terms. Many old tools still continue to work just fine after
> > a decade or more.
>
> Yes, I think I agree with Luca's conclusion, but not so much with this
> argument: the parts of networking that are relevant for a default choice
> that lets users get started (approximately the subset supported by d-i)
> don't move that fast.

Eh, ish. The kind of bugs and weirdness that regularly come through
because of what changes in the kernel paint a different picture to me,
even for the really basic stuff. Just to take two of the most recent
ones:

https://github.com/systemd/systemd/issues/33104
https://github.com/systemd/systemd/issues/29701

"laptop with ethernet port attached to a dock with another ethernet
port" and "ipv6 enabled with privacy extension" do not look like
abstruse corner cases to me, they look pretty basic. And yet...

This is not an argument against netplan of course, given it's a
configuration layer on top of networkd/network-manager. It is an
argument against ifupdown though.

And then once you start adding more complex things like bonding this
just goes out of the window. Yes the installer doesn't allow
configuring bonding (thank ), but the default choice goes beyond
what the installer allows to configure in the installation phase.

Re: what about Netplan?

[Luca Boccassi]

On Sun, 14 Jul 2024 at 19:21, Simon McVittie  wrote:
> Dependency size and maintenance
> ---
>
> I also notice that the netplan.io package would bring GLib, Python,
> python3-dbus and python3-yaml into the dependencies of the base system,
> among others. As an upstream and downstream maintainer of GLib, and in
> practice the only upstream and downstream unmaintainer[1] of python3-dbus,
> I'm not comfortable with that, both for size reasons (GLib and Python
> are not small) and for quality and maintainedness reasons (python3-dbus).
>
> GLib and Python are relatively large in both on-disk size and attack
> surface, but they're essential to our desktop installations anyway,
> reinventing them in parallel would be worse than reusing them, and
> their maintainers have already accepted the responsibility of them
> being an important system component. So those two are actually not my
> main concern here, and if we had no good alternative, I'd be saying:
> yes they're relatively bulky for a container/embedded use-case, but on
> a typical server installation they're fine.

Let's put some hard numbers on the table given this is an important
detail. The following is all starting from a default debootstrapped
unstable.

With networkd only we can drop ifupdown, net changes:

REMOVING:
  ifupdown

Summary:
  Upgrading: 0, Installing: 0, Removing: 1, Not Upgrading: 0
  Freed space: 207 kB


Using network-manager in headless mode (no GUI) brings in:

Installing:
  network-manager

Installing dependencies:
  ca-certificates  libgudev-1.0-0libnl-genl-3-200
 libssh2-1t64
  dbus libicu72  libnl-route-3-200
 libteamdctl0
  dbus-bin libjim0.82t64 libnm0
 libusb-1.0-0
  dbus-daemon  libldap-2.5-0 libpam-systemd
 libxml2
  dbus-session-bus-common  libldap-commonlibpcap0.8t64
 modemmanager
  dbus-system-bus-common   libmbim-glib4 libpcsclite1
 openssl
  dbus-user-sessionlibmbim-proxy
libpolkit-agent-1-0polkitd
  dns-root-datalibmbim-utils libpolkit-gobject-1-0  ppp
  dnsmasq-base libmm-glib0   libpsl5t64
 publicsuffix
  libbluetooth3libndp0   libqmi-glib5
 sgml-base
  libbrotli1   libnetfilter-conntrack3   libqmi-proxy
 shared-mime-info
  libcurl3t64-gnutls   libnfnetlink0 libqmi-utils
 usb-modeswitch
  libdbus-1-3  libnghttp2-14 libqrtr-glib0
 usb-modeswitch-data
  libduktape207libnghttp3-9  librtmp1
 wireless-regdb
  libexpat1libngtcp2-16  libsasl2-2
 wpasupplicant
  libglib2.0-0t64  libngtcp2-crypto-gnutls8  libsasl2-modules
 xdg-user-dirs
  libglib2.0-data  libnl-3-200
libsasl2-modules-dbxml-core

Suggested packages:
  low-memory-monitor  libsasl2-modules-gssapi-mit
libsasl2-modules-sql  comgt debhelper
  libcryptsetup12 | libsasl2-modules-gssapi-heimdal  libteam-utils
wvdial
  libtss2-rc0t64  libsasl2-modules-ldap  iptables
wpagui
  pcscd   libsasl2-modules-otp   sgml-base-doc
libengine-pkcs11-openssl

Summary:
  Upgrading: 0, Installing: 69, Removing: 0, Not Upgrading: 0
  Download size: 28.2 MB
  Space needed: 110 MB / 8295 MB available


Installing netplan.io instead brings in:

Installing:
  netplan.io

Installing dependencies:
  ca-certificates  libexpat1  libsqlite3-0
 python3-gi python3-yaml
  dbus libgirepository-1.0-1  libxml2
 python3-markdown-itpython3.12
  dbus-bin libglib2.0-0t64libyaml-0-2
 python3-mdurl  python3.12-minimal
  dbus-daemon  libglib2.0-datamedia-types
 python3-minimalshared-mime-info
  dbus-session-bus-common  libicu72   netplan-generator
 python3-netifaces  xdg-user-dirs
  dbus-system-bus-common   libnetplan1openssl
 python3-netplan
  gir1.2-girepository-2.0  libpython3-stdlib  python3
 python3-pkg-resources
  gir1.2-glib-2.0  libpython3.12-minimal  python3-cffi-backend
 python3-pygments
  libdbus-1-3  libpython3.12-stdlib   python3-dbus
 python3-rich

Suggested packages:
  default-dbus-session-bus  | wpasupplicant python3-tk
python-pygments-doc  binutils
  | dbus-session-busopenvswitch-switch  python3-venv
ttf-bitstream-vera   binfmt-support
  low-memory-monitoriw  python-dbus-doc
python3.12-venv
  network-manager   python3-doc python3-setuptools
python3.12-doc

Summary:
  Upgrading: 0, Installing: 42, Removing: 0, Not Upgrading: 0
  Download size: 25.2 MB
  Space needed: 101 MB / 8340 MB available


So we have a net gain of ~200K when using networkd, a net loss of
~110M whe

Re: what about Netplan?

[Luca Boccassi]

On Mon, 15 Jul 2024 at 14:36, Lukas Märdian  wrote:
> = "How to do networking on Debian?" =
>
> If we have to tell our users and sysadmins to do "X" on Debian server systems
> (using ifupdown or potentially sd-networkd), while doing "Y" on Debian desktop
> systems (using NetworkManager), while doing "Z" on Debian cloud systems (using
> Netplan), while doing something totally different on RaspberryPi (or alike) 
> boards
> that run a Debian server setup, but using WiFi as their primary network 
> interface,
> that's just a really bad user experience.
>
> Using Debian should NOT feel like using different distros. And we really need 
> a
> common way to do network configuration. With Netplan we can tell people to 
> just use
> use the "dhcp4: true" setting (for example), which will work on all Debian 
> systems
> and is automatically translated to the corresponding backend for
> server/desktop/cloud/embedded usecases.
>
> All while giving sysadmins the flexibilty to fully utilize the underlying 
> network
> daemon directly, if they feel like writing native configuration for it (or 
> don't
> like Netplan).

Assuming that's really needed, and it's far from clear that different
use cases should really use the exact same things, using
network-manager everywhere would achieve the exact same result,
without pulling in additional dependencies, and without being tied to
the internal decisions made in Canonical that we cannot do anything to
influence. Again, not your fault, but existing examples don't exactly
inspire a lot of confidence in that regard: mir, upstart, unity,
lxd...

Re: what about Netplan?

[Luca Boccassi]

On Tue, 16 Jul 2024 at 00:26, Steve Langasek  wrote:
>
> On Mon, Jul 15, 2024 at 03:47:16PM +0100, Luca Boccassi wrote:
> > Assuming that's really needed, and it's far from clear that different
> > use cases should really use the exact same things, using
> > network-manager everywhere would achieve the exact same result,
> > without pulling in additional dependencies, and without being tied to
> > the internal decisions made in Canonical that we cannot do anything to
> > influence. Again, not your fault, but existing examples don't exactly
> > inspire a lot of confidence in that regard: mir, upstart, unity,
> > lxd...
>
> You could compile a similar list of software projects that were abandoned
> when Red Hat stopped funding them.  Or of entirely community-backed free
> software projects that are moribund.  I think it's prejudicial to argue that
> a piece of free software should not be adopted because its development is
> funded by a company which, over the course of 20 years, has made strategic
> decisions to discontinue investments in other, unrelated projects.

Yes, you could compile those lists, and those items would be
problematic too. That's the point made in
https://lists.debian.org/debian-devel/2024/07/msg00137.html and I
think it's spot-on.

> Either netplan is technically sound, providing a sensible configuration
> language that meets the needs of Debian users and has a high-quality code
> base, in which case it should not actually be a problem for Debian to
> maintain it in the event that Canonical discontinues work on it; or it
> isn't, and we can stop the discussion there.

Again, yes, in that case it would most definitely be a problem, as
explained in https://lists.debian.org/debian-devel/2024/07/msg00137.html
being saddled with having to maintain a discontinued product would
absolutely be an issue, which is enough to make it a less prefereable
solution, independently of whatever specific technical merits it might
or might not have. In the end all these projects largely just work, in
the sense of allowing to successfully achieve the goal of configuring
network interfaces, so in this case other individual metrics are less
important than other factors such as, how many other distributions are
using it and how resilient is maintenance against shift in priorities
of a single entity, which translates to, how many active maintainers
and from which backgrounds are working on the project.

Re: what about Netplan?

[Luca Boccassi]

On Tue, 16 Jul 2024 at 14:05, gregor herrmann  wrote:
>
> On Tue, 16 Jul 2024 13:13:16 +0200, Philip Hands wrote:
>
> > I suspect having something that's agnostic about the underlying
> > implementation as our default would be rather better for the non-systemd
> > options that people care about, …
> > Also, networkd doesn't support non-Linux and non-systemd systems AFAIK,
>
> AFAICS, the netplan packages (netplan.io, netplan-generator) currently
> have a hard dependency on systemd.

And that is fine - it's entirely appropriate for any optional or
default to do so. One wishing to "experiment with other systems" can
simply build an image that only contains ifupdown, and that will work
as expected, regardless of what the default is. Such a workflow is
needed for ports like Hurd, for example.

Re: Size measuring contest (Was: what about Netplan?)

[Luca Boccassi]

On Tue, 16 Jul 2024 at 14:46, Daniel Gröber  wrote:
>
> Hi Luca,
>
> On Mon, Jul 15, 2024 at 02:50:17PM +0100, Luca Boccassi wrote:
> > Let's put some hard numbers on the table given this is an important
> > detail. The following is all starting from a default debootstrapped
> > unstable.
> >
> > With networkd only we can drop ifupdown, net changes:
> >
> > REMOVING:
> >   ifupdown
> >
> > Summary:
> >   Upgrading: 0, Installing: 0, Removing: 1, Not Upgrading: 0
> >   Freed space: 207 kB
> >
> >
> > Using network-manager in headless mode (no GUI) brings in:
> > [...]
> >
> > Summary:
> >   Upgrading: 0, Installing: 69, Removing: 0, Not Upgrading: 0
> >   Download size: 28.2 MB
> >   Space needed: 110 MB / 8295 MB available
> >
> >
> > Installing netplan.io instead brings in:
> > [...]
> >
> > Summary:
> >   Upgrading: 0, Installing: 42, Removing: 0, Not Upgrading: 0
> >   Download size: 25.2 MB
> >   Space needed: 101 MB / 8340 MB available
> >
> >
> > So we have a net gain of ~200K when using networkd, a net loss of
> > ~110M when using network-manager, and a net loss of ~101M when using
> > netplan.
>
> For completeness let me turn that arguemnt around on you as systemd
> maintainer. Why is systemd-networkd, a component currently disabled by
> default mind you bloating our base system? :)

Because it is used commonly enough in servers/containers, and doesn't
bring in any additional dependencies

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Luca Boccassi]

On Thu, 1 Aug 2024 at 10:36, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi Otto,
>
> Quoting Otto Kekäläinen (2024-07-28 00:38:40)
> > I have drafted a new DEP at
> > https://salsa.debian.org/dep-team/deps/-/merge_requests/8 titled "DEP-18:
> > Enable true open collaboration on all Debian packages".
> >
> > Direct link to raw text:
> > https://salsa.debian.org/dep-team/deps/-/raw/798bfa5a1e1609afd4e48ee20aff80a82bcd4a2f/web/deps/dep18.mdwn
> >
> > This would have been a great topic to discuss in person at DebConf, but
> > unfortunately I can't attend this year, so I'll just kick this off on the
> > mailing list.
>
> thank you for working on this.
>
> > This is continuation to the 'single maintainership' discussions earlier this
> > year. I also think that more unified and open collaboration processes could
> > help decrease maintainer burnout, lower barrier for existing and new
> > maintainers to help with multiple packages, and overall perhaps also improve
> > quality of uploads by having more attention on the source code prior to
> > upload.
>
> A slightly related topic: what is everybody's opinion on maybe starting with
> some of the basic items of DEP-18, lets say items 1-3, and prescribe them to
> only a very small set of packages in Debian. And which set of packages in
> Debian should *especially* have easy open collaboration enabled if not the set
> of source packages producing our Essential:yes packages? So why not start with
> the source packages in the Essential:yes set, have them adhere to better
> collaboration standards like packaging in git on salsa and with salsa CI
> enabled?

This is a great idea, as it would provide a real testing ground while
having a clearly delineated and well defined scope, and with the
greatest potential return of investment.

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Luca Boccassi]

On Thu, 1 Aug 2024 at 09:49, Jonas Smedegaard  wrote:
> Quoting Otto Kekäläinen (2024-08-01 07:30:18)
> > You have for sure developed an optimal workflow for yourself.
>
> Then I have failed: I have strived towards a collaborative workflow.
>
> Just not a web-centered collaborative workflow, but an email- and chat-
> based one.

Emails are actually a barrier against collaboration, and actively
hinder it by preventing new people from joining in. Please understand
that, outside of the demographic group of people who started using
computers in the 70s and 80s, the vast majority of opinions on the
topic of email as a collaborative tool range from "barely tolerated"
to "deeply hated". Emails sucks, it's horrible and outdated stuff
based on horrible and outdated procolos and ideas, and today it is by
and large a system to deliver spam, phishing and malware, with
occasional barely useful things like confirming registration on a new
website or resetting a lost password. The vast majority of people who
are forced to use emails do so for work via a
work-mediated/administered interface that tries to make it somewhat
tolerable, like Outlook or Gmail. By and large people born this side
of the millennium look at people running their own email workflows as
you would look at someone programming with punchcards - an interesting
curiosity if found in a museum re-enactment or a history book
photograph, and a strong reason to get the heck out of there
otherwise. And outside of the tech bubble it's even worse. Some
projects like the kernel can afford to let the curmudgeons continue to
dictate its use because it's a de-facto monopoly and if one wants to
get things merged in Linux there is no alternative, and even then
there are continuous grumblings and attempts by the IT infra managers
to build bridges with the 21st century with custom and bespoke
kernel-specific kludges, to the point where some subtree maintainers
have just gone "fuck this" and set up their own Gitlab repos for their
own subtrees. We don't have that luxury: there are plenty of
alternatives to Debian that work just as well for the same use cases.

The number of people involved in programming, software engineering, IT
and any other related field has absolutely exploded since the 90s. The
number of Debian project members has remained pretty much flat at
~1000 people, and we just about manage to backfill retirements/MIAs.
To pick a random example, a less well known, less used, less popular
distribution like Nixos has 7000+ contributors listed on Github. It
would be wise to stop and reflect on why this is the case every now
and then. The passage of time and demographics are cruel and
merciless.

And the fact that, two decades or so after it became standard practice
in the industry, there is _still_ resistance against having CI runs
_before_ pushing things out simply boggles my mind. This is a no
brainer: run a CI before uploading, even a very basic one is just
fine, better than nothing. It's 2024, "but but but the Gitlab UI
doesn't display nicely on my 80x24 green phosphor monochrome monitor"
just doesn't cut it anymore, sorry.

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Luca Boccassi]

On Thu, 1 Aug 2024 at 13:43, Charles Plessy  wrote:
>
> Le Thu, Aug 01, 2024 at 12:23:43PM +0100, Luca Boccassi a écrit :
> >
> > run a CI before uploading, even a very basic one is just fine, better
> > than nothing.
>
> Thanks for the remider !  I will have a closer look at Salsa CI instead
> of trying to understand how to run autopkgtests locally.
>
> Would there be a way to get Salsa upload and tag the package if the CI
> tests pass and the changelog signals a release?  Or does somebody has a
> script which can screen a Salsa group for ready uploads, and run clone
> && buildpackage && dput automatically ?

See https://www.debian.org/vote/2024/vote_002 and the associated email
threads on debian-vote

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Luca Boccassi]

On Thu, 1 Aug 2024 at 18:16, Marc Haber  wrote:
>
> On Thu, 1 Aug 2024 12:23:43 +0100, Luca Boccassi 
> wrote:
> >The vast majority of people who
> >are forced to use emails do so for work via a
> >work-mediated/administered interface that tries to make it somewhat
> >tolerable, like Outlook or Gmail.
>
> This must be the least accurate statement about Outlook that I have
> ever read. Outlook does not have a single feature that makes email
> more tolerable in any way. Au contraire. The feature called threading
> is incompatible with the rest of the world and not even very useful in
> an outlook-only environment, the search function finds everything but
> the mail you're looking for, and Outlook's disability to quote
> decently is notorious for having led to the whole world generating
> only top-posting replies.
>
> Outlook is essentially responsible for making email so much worse
> today than it was in the 1990s.
>
> Even Salsa's and github's praised way to communitate in an MR is
> VASTLY inferior to a decently threading mail client like mutt or even
> Thunderbird.
>
> I violently disagree with the rest of the message as well and am not
> willing to spoil the rest of my day by replying in detail. I'd prefer
> learning a bit more Slowfoxtrot later tonight.

Again, please understand that outside of the bubble of tech nerds of
the 70s/80s, saying out loud phrases such as "The only right way to
collaborate is reading and writing emails is in my terminal" means
getting looked at like being a dinosaur just escaped from a museum.
The rest of the universe just doesn't work like that, sorry. There's
nothing wrong with being a dinosaur for an individual of course, we
will all become one at some point, but optimizing for making dinosaurs
happy from the simple perspective of demographics is a sure way for a
project to slowly slide into irrelevance. The fact that the project
membership has just about managed to remain flat while the tech sector
absolutely exploded in size should send shivers down everyone's
spines.

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Luca Boccassi]

On Thu, 1 Aug 2024 at 21:57, Marc Haber  wrote:
>
> On Thu, 1 Aug 2024 18:36:29 +0100, Luca Boccassi 
> wrote:
> >On Thu, 1 Aug 2024 at 18:16, Marc Haber  wrote:
> >>
> >> On Thu, 1 Aug 2024 12:23:43 +0100, Luca Boccassi 
> >> wrote:
> >> >The vast majority of people who
> >> >are forced to use emails do so for work via a
> >> >work-mediated/administered interface that tries to make it somewhat
> >> >tolerable, like Outlook or Gmail.
> >>
> >> This must be the least accurate statement about Outlook that I have
> >> ever read. Outlook does not have a single feature that makes email
> >> more tolerable in any way. Au contraire. The feature called threading
> >> is incompatible with the rest of the world and not even very useful in
> >> an outlook-only environment, the search function finds everything but
> >> the mail you're looking for, and Outlook's disability to quote
> >> decently is notorious for having led to the whole world generating
> >> only top-posting replies.
> >>
> >> Outlook is essentially responsible for making email so much worse
> >> today than it was in the 1990s.
> >>
> >> Even Salsa's and github's praised way to communitate in an MR is
> >> VASTLY inferior to a decently threading mail client like mutt or even
> >> Thunderbird.
> >>
> >> I violently disagree with the rest of the message as well and am not
> >> willing to spoil the rest of my day by replying in detail. I'd prefer
> >> learning a bit more Slowfoxtrot later tonight.
> >
> >Again, please understand that outside of the bubble of tech nerds of
> >the 70s/80s, saying out loud phrases such as "The only right way to
> >collaborate is reading and writing emails is in my terminal" means
> >getting looked at like being a dinosaur just escaped from a museum.
> >The rest of the universe just doesn't work like that, sorry. There's
> >nothing wrong with being a dinosaur for an individual of course, we
> >will all become one at some point, but optimizing for making dinosaurs
> >happy from the simple perspective of demographics is a sure way for a
> >project to slowly slide into irrelevance. The fact that the project
> >membership has just about managed to remain flat while the tech sector
> >absolutely exploded in size should send shivers down everyone's
> >spines.
>
> Now that you have added personal insult while not adding anything for
> the cause, I recuse myself from continuing this situation. I am happy
> that I don't need to collaborate with you in my Debian efforts.

What are you on about? What personal insult?

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Luca Boccassi]

On Thu, 1 Aug 2024 at 18:23, Jeremy Stanley  wrote:
>
> On 2024-08-01 12:23:43 +0100 (+0100), Luca Boccassi wrote:
> [...]
> > To pick a random example, a less well known, less used, less
> > popular distribution like Nixos has 7000+ contributors listed on
> > Github.
> [...]
>
> Just to pick on this particular point, because I see this metric
> used all the time by projects trying to inflate public impressions
> of their size: you're aware that GitHub counts someone as a
> "contributor" even if all the do is leave a comment on a bug report,
> right? By that gauge, Debian is probably orders of magnitude larger.

I'm not, no, I thought it was committers/authors? But I haven't really
looked it up so you might very well be right. Where did you find that
defined?

Re: Handling of entropy during boot

[Luca Boccassi]

On Wed, 2019-01-16 at 11:05 +0100, Guido Günther wrote:
> Hi,
> On Mon, Jan 14, 2019 at 05:56:20PM +0100, W. Martin Borgert wrote:
> > Quoting Michael Stone :
> > > Unless the cpu supports rdrand/rdseed, installing rng-tools5
> > > won't
> > > really change anything. If it does support those, it probably
> > > makes more
> > > sense going forward to just enable CONFIG_RANDOM_TRUST_CPU rather
> > > than
> > > installing another package.
> > 
> > This option is only available for some architectures (X86, S390,
> > PPC)?
> > What about the others?
> 
> There's also jitterentropy-rngd which does the trick but I haven't
> looked at the security implications.
>  -- Guido

FWIW I've been using jitterentropy-rngd and rng-tools in production for
years, in Azure/VMWare/AWS x86 VMs, exactly for this problem. Haven't
been hacked so far... as far as I know :-)

-- 
Kind regards,
Luca Boccassi

signature.asc
Description: This is a digitally signed message part

Bug#930414: ITP: python-applicationinsights -- Application Insights API for Python

[Luca Boccassi]

Package: wnpp
Severity: wishlist
Owner: "Luca Boccassi" 
X-Debbugs-CC: debian-devel@lists.debian.org
Control: block 930413 by -1

* Package name: python-applicationinsights
  Version : 0.11.9
  Upstream Author : Microsoft Corporation
* URL : https://github.com/Microsoft/ApplicationInsights-Python
* License : MIT
* Programming Lang: Python
* Description : Application Insights API for Python

This project extends the Application Insights API surface to support
Python. Application Insights is a service that allows developers to
keep their application available, performing and succeeding. This
Python module will allow you to send telemetry of various kinds (event,
trace, exception, etc.) to the Application Insights service where they
can be visualized in the Azure Portal.

This package was previously uploaded by Iain R. Learmonth via #838717,
but subsequently removed via #918546 due to lack of time.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#930415: ITP: vsts-cd-manager -- Visual Studio Team Services Continuous Delivery Manager

[Luca Boccassi]

Package: wnpp
Severity: wishlist
Owner: "Luca Boccassi" 
X-Debbugs-CC: debian-devel@lists.debian.org
Control: block 930413 by -1

* Package name: vsts-cd-manager
  Version : g2649d23
  Upstream Author : Microsoft Corporation
* URL : https://github.com/microsoft/vsts-cd-manager
* License : MIT
* Programming Lang: Python
* Description : Visual Studio Team Services Continuous Delivery Manager

This project provides the class ContinuousDeliveryManager and
supporting classes for Azure. This CD manager class allows the caller
to manage Azure Continuous Delivery pipelines that are maintained
within a VSTS account.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Bug#930416: ITP: azure-cosmos-python -- Azure Cosmos DB Python SDK

[Luca Boccassi]

Package: wnpp
Severity: wishlist
Owner: "Luca Boccassi" 
X-Debbugs-CC: debian-devel@lists.debian.org
Control: block 930413 by -1

* Package name: azure-cosmos-python
  Version : 2.3.3
  Upstream Author : Microsoft Corporation
* URL : https://github.com/Azure/azure-cosmos-python
* License : MIT
* Programming Lang: Python
* Description : Azure Cosmos DB Python SDK

This project implements Python classes for interacting with Azure
Cosmos SQL DB, and is a dependency of azure-cli.

Version 2.3.3 will be uploaded rather than the latest 3.x, as that's
what azure-cli needs. Once azure-cli moves to the latest series (there
was a backward-incompatible namespace change), I will update it.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: Missing package in Debian Buster "evolution-ews"

[Luca Boccassi]

On Mon, 2019-06-17 at 11:13 +0200, Bas Couwenberg wrote:
> On 2019-06-17 09:56, Ronald Jaeckel wrote:
> > The only package I've missed was evolution-ews. It's not in the
> > repositories anymore. So I took the package from sid and it worked
> > for
> > me. My question is: Will be the package evolution-ews integrated in
> > Buster before the official release?
> 
> No, the package is blocked from transitioning to testing for quite a 
> while, see:
> 
>   
> https://tracker.debian.org/pkg/evolution-ews
> 
> 
> When the package gets fixed, it may be possible to backport it to 
> buster-backports. It wasn't backported before, so the maintainer is 
> unlikely to do that.
> 
> Perhaps you can get involved in resolving the issues in the package
> to 
> get it into testing, and maintain the backports once it's in testing.
> 
> Kind Regards,
> 
> Bas

Hi,

I rely on evolution-ews at $work so I opened 2 PRs on Salsa:

https://salsa.debian.org/gnome-team/evolution-data-server/merge_requests/1
https://salsa.debian.org/gnome-team/evolution-ews/merge_requests/2

Tested the changes, seem to work fine with the instance we use at
$work. It would be great if we could get the plugin back into Buster
before the release.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Fri, 21 Apr 2023 at 13:16, Raphael Hertzog  wrote:
>
> Hello,
>
> On Mon, 03 Apr 2023, Helmut Grohne wrote:
> > Please consider it to be a piece of best
> > intentions at reconciling feedback wherever I could. At the time of this
> > writing it certainly is not consensus, but consensus is what I seek
> > here.  Without further ado, the full DEP text follows after my name
> > while it also is available at
> > https://salsa.debian.org/dep-team/deps/-/merge_requests/5
>
> I'd like to express some disappointment that nobody replied publicly
> sofar. Last year's developer survey concluded that "Debian should complete
> the merged-/usr transition" was the most important project for Debian [1]
> (among those proposed in the survey). That's what we are trying to do
> here and it would be nice to build some sort of consensus on what it means
> in terms of changes for dpkg.
>
> I know that Guillem (dpkg's maintainer) is generally opposed to the
> approach that Debian has followed to implement merged-/usr but I have
> yet to read his concerns on the changes proposed here

There was an answer from the maintainer, 2 weeks ago:
https://lists.debian.org/debian-dpkg/2023/04/msg1.html
Essentially, the answer was "no", so...

After Bookworm ships I plan to propose a policy change to the CTTE and
policy maintainers to forbid shipping files in the legacy directories
altogether, followed by a debhelper change to adjust any stragglers
automatically at build time and a mass rebuild, plus MBF for the small
% that does not use dh and a piuparts test to stop migration for
anything that is uploaded and doesn't comply. That should bring the
matter to an end, without needing to modify dpkg.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 22 Apr 2023 at 11:41, Simon McVittie  wrote:
>
> On Fri, 21 Apr 2023 at 15:29:33 +0100, Luca Boccassi wrote:
> > After Bookworm ships I plan to propose a policy change to the CTTE and
> > policy maintainers to forbid shipping files in the legacy directories
> > altogether, followed by a debhelper change to adjust any stragglers
> > automatically at build time and a mass rebuild
>
> That seems quite likely to trigger the scenario Helmut is trying to avoid,
> which if I understand correctly is this:
>
> * foo_12.0 in Debian 12 ships /lib/abcd
> * bar_13.0 takes over /lib/abcd from foo, but because of either your
>   proposed change or a manual action by the maintainer, it is actually in
>   the data.tar as ./usr/lib/abcd (not ./lib/abcd like it was in foo_12.0)
> * the maintainer of bar didn't add the correct Breaks/Replaces on foo
> * a user upgrading from Debian 12 to 13 installs bar_13.0, perhaps pulled
>   in as a dependency
> * expected result: dpkg refuses to unpack bar ("trying to overwrite ..."),
>   the upgrade is cancelled, and the user reports a RC bug in bar
> * actual result: /usr/lib/abcd in bar quietly overwrites /lib/abcd from foo
> * if bar is subsequently removed, then dpkg (and therefore apt) thinks foo
>   is fully functional, but in fact /{usr/,}lib/abcd is missing
>
> (For simplicity I've described that scenario in terms of files directly
> shipped in the data.tar, but dpkg also tracks the ownership of files
> created by dpkg-divert or alternatives, and similar things can happen
> to those.)
>
> I had hoped that the last section of technical committee resolution
> #994388 (which concerns this situation) would become irrelevant in Debian
> 13, but it's looking as though without the sort of dpkg changes discussed
> in this thread, the concern about files moving between packages would
> remain a valid concern.
>
> However, as far as I can see, the other reasons not to do this that were
> mentioned in the last section of #994388 *do* become irrelevant in Debian
> 13, so solving the files-moved-between-packages thing is the last major
> blocker for doing what you propose. (Unless someone has a reason why this
> is not the case?)
>
> You might reasonably say that "the maintainer of bar didn't add the
> correct Breaks/Replaces on foo" is a RC bug in bar - and it is! - but
> judging by the number of "missing Breaks/Replaces" bug reports that have
> to be opened by unstable users (sometimes me), it's a very easy mistake
> to make.
>
> One thing that's particularly tricky about this is that the move from
> / into /usr and the move from foo to bar might be 18 months apart if
> they happen to occur at opposite ends of our stable release cycle. In
> particular, if the move from / into /usr is done as soon as the Debian 13
> cycle opens, we cannot predict whether the packages that have undergone
> that move will also need to undergo a package split/merge at some point
> in the following 18 months (but it's reasonable to assume that at least
> some of them will).

We already have piuparts tests detecting files moving, it should be
easy enough to extend that to check that the appropriate
Breaks/Replaces have been added. Correct me if I'm wrong, but I
believe it's already against policy to do this without
Breaks/Replaces, so it's not a use case that we need to support, no?
If someone does that by mistake, the package will not migrate to
testing.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 22 Apr 2023 at 11:50, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Fri, Apr 21, 2023 at 03:29:33PM +0100, Luca Boccassi wrote:
> > After Bookworm ships I plan to propose a policy change to the CTTE and
> > policy maintainers to forbid shipping files in the legacy directories
> > altogether, followed by a debhelper change to adjust any stragglers
> > automatically at build time and a mass rebuild, plus MBF for the small
> > % that does not use dh and a piuparts test to stop migration for
> > anything that is uploaded and doesn't comply. That should bring the
> > matter to an end, without needing to modify dpkg.
>
> I agree with the goal of removing aliases by moving files to their
> canonical locations. However, I do not quite see us getting there in the
> way you see it, but maybe I am missing something. As long as dpkg does
> not understand the effects of aliasing, we cannot safely move those
> files and thus the file move moratorium will have to be kept in place.
> And while moving the files would bring the matter to an end, we cannot
> do so without either modifying dpkg or rolling back the transition and
> starting over. I hope that we all agree that rolling back would be too
> insane to even consider, but I fail to see how you safely move files
> without dpkg being changed. Can you elaborate on that aspect?

Moving files within _the same_ package is actually fine as far as I
know. It's moving between location _and_ packages within the same
upgrade that is problematic. The piuparts test I added is overzealous,
but it doesn't need to be.

> I'd also be interested on how you plan to move important files in
> essential packages. This is an aspect raised by Simon Richter and where
> I do not see an obvious answer yet.

Do you have a pointer? Not sure I follow what "important" files means
here, doesn't ring a bell.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Tue, 25 Apr 2023 at 20:07, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Sat, Apr 22, 2023 at 01:06:18PM +0100, Luca Boccassi wrote:
> > On Sat, 22 Apr 2023 at 11:50, Helmut Grohne  wrote:
> > > On Fri, Apr 21, 2023 at 03:29:33PM +0100, Luca Boccassi wrote:
> > > > After Bookworm ships I plan to propose a policy change to the CTTE and
> > > > policy maintainers to forbid shipping files in the legacy directories
> > > > altogether, followed by a debhelper change to adjust any stragglers
> > > > automatically at build time and a mass rebuild, plus MBF for the small
> > > > % that does not use dh and a piuparts test to stop migration for
> > > > anything that is uploaded and doesn't comply. That should bring the
> > > > matter to an end, without needing to modify dpkg.
> > >
> > > I agree with the goal of removing aliases by moving files to their
> > > canonical locations. However, I do not quite see us getting there in the
> > > way you see it, but maybe I am missing something. As long as dpkg does
> > > not understand the effects of aliasing, we cannot safely move those
> > > files and thus the file move moratorium will have to be kept in place.
> > > And while moving the files would bring the matter to an end, we cannot
> > > do so without either modifying dpkg or rolling back the transition and
> > > starting over. I hope that we all agree that rolling back would be too
> > > insane to even consider, but I fail to see how you safely move files
> > > without dpkg being changed. Can you elaborate on that aspect?
> >
> > Moving files within _the same_ package is actually fine as far as I
> > know. It's moving between location _and_ packages within the same
> > upgrade that is problematic. The piuparts test I added is overzealous,
> > but it doesn't need to be.
>
> You got me interested to dig deeper. I looked into that piuparts
> check[1]. From what I understand, it does something differently from
> what you suggest here. It detects files moved between / and /usr (which
> is what is going to happen according to your plan) and it does not
> detect files being moved between packages (which would actually be
> problematic here). It also does not produce an error (merely a warning),
> so it doesn't halt testing migration and in particular, it doesn't
> detect the problematic situation at all. That's kinda disappointing.

It was a while ago, but from my recollection it did check both things,
and it did fail the run. I'll check again when I have a moment (or
better: MRs welcome ;-) ).
Did you run it with any specific cases/packages to check this?

> You also side stepped the question of how to handle the situation where
> we've moved files from / to /usr and then need to move files between
> packages in a safe way, though your other response to Simon McVittie
> suggests you have an idea there.
>
> On Sat, 22 Apr 2023 13:03:01 +0100, Luca Boccassi wrote:
> > We already have piuparts tests detecting files moving, it should be
> > easy enough to extend that to check that the appropriate
> > Breaks/Replaces have been added. Correct me if I'm wrong, but I
> > believe it's already against policy to do this without
> > Breaks/Replaces, so it's not a use case that we need to support, no?
> > If someone does that by mistake, the package will not migrate to
> > testing.
>
> Yeah, we agree that you need Breaks+Replaces. The issue here is that due
> to dpkg not knowing about the aliasing, Breaks+Replaces is insufficient.
> Due to the insufficiency the CTTE enacted the moratorium.
>
> My impression is that you believe that with bookworm, the moratorium is
> being lifted and thus we can start moving files. Unfortunately, the
> underlying problem does not go away just because we've released
> bookworm. It's a problem that is unique to merged installations and
> those are not going to go away in bookworm.

Not quite, from my point of view I don't see a problem with keeping
the moratorium in place for bookworm, personally.
In practice, it would mean it applies to everything that currently
ships files in bin/sbin/lib, as all of those are moved automatically
and thus can no longer move to another package at the same time, which
is of course a non-trivial constraint.
But it sounds like you came up with an even better way:

> So yeah, we all want these files moved to their canonical locations and
> I kinda like the simplicity of your approach, but thus far my
> understanding is that it is plain broken and doesn't work. Well, yeah it
> does work in the sense that we break user installations durin

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Wed, 26 Apr 2023 at 10:11, Simon Richter  wrote:
>
> Hi,
>
> On Tue, Apr 25, 2023 at 09:07:28PM +0200, Helmut Grohne wrote:
>
> > This and /bin/sh is the kind of files I'd consider important. And then
> > upon thinking further it became more and more difficult for me to make
> > sense of the objection. On a merged system, we can just move that file
> > to its canonical location without having any trouble even with an
> > unmodified dpkg. So from my pov, the question about important files can
> > be disregarded. I hope Simon Richter agrees.
>
> Yes, the relevant code at
>
> https://github.com/guillemj/dpkg/blob/main/src/main/unpack.c#L749
>
> already handles moving a file inside the same package, and that has
> existed for some time, that's why I use two packages for the PoC.
>
> I have not looked for more issues beyond that, so there might be others
> lurking in the depths of this code.
>
> What I'm mostly concerned about (read: have not verified either way)
> with /lib/ld.so and /bin/sh is what happens when dpkg learns of /bin and
> /lib as symlinks -- because right now, the symlinks created by usrmerge
> are protected by the rule that if dpkg expects a directory and finds a
> symlink, that is fine because that is obviously an action taken by the
> admin.
>
> But if dpkg sees a package containing these as symlinks, then this is
> entered into the dpkg database, and subject to conflict resolution, and
> for that, a separate rule exists that directory-symlink conflicts are
> resolved in favour of the directory, so the interaction between a newer
> base-files packages shipping /lib as a symlink and an older or
> third-party package containing /lib as a directory (e.g. a kernel
> package from a hosting provider) could overwrite the /lib symlink.
>
> It might be possible to avoid that by never shipping /lib as a symlink
> and always creating it externally, but I still think that's kind of
> wobbly.

IMHO we should not ship the top-level symlinks in a package. The
reason for that is to allow the use case where /usr is a separate
vendor partition and / is either a luks volume or a tmpfs, and thus
the top-level symlinks are ephemeral and re-created on boot on the
fly. If they were part of a package, that would get awkward to say the
least.
I really would like to move toward the direction of having exclusively
/usr and /etc shipped in data.tar, and everything else created locally
as needed, and that includes files in /.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

> unstable
> diversion of /bin/zegrep to /bin/zegrep.gzip by zutils in stable, testing, 
> unstable
> diversion of /bin/zfgrep to /bin/zfgrep.gzip by zutils in stable, testing, 
> unstable
> diversion of /bin/zgrep to /bin/zgrep.gzip by zutils in stable, testing, 
> unstable
>
> All other diversion affect /etc or /usr and I think we're not going to
> move any files from /usr to /. So this is a complete list as of today
> and I have to say, I expected it to be longer. In effect, we're talking
> about merely 8 packages.
>
> For completeness sake, I also looked at the other packages mentioning
> dpkg-divert in their preinst to catch false negatives. I'll skip
> diversions inside /usr as well as removals of diversions here:
>  * amazon-ec2-net-utils: diversion inside /etc
>  * angband: comment about diversions
>  * arpwatch: comment about diversions
>  * dash: complex use of conditional diversions via postinst
>  * dist: comment about diversions
>  * gpr: conditional diversion (inside /usr)
>  * iputils-arping: check for an existing diversion
>  * iputils-clockdiff: check for an existing diversion
>  * iputils-ping: check for an existing diversion
>  * ld10k1: comment about diversions
>  * mailagent: comment about diversions
>  * oping: checks for an existing diversion
>  * psgml: comment about diversions
>  * ucf: comment about diversions
>  * wireshark-common: checks for an existing diversion

This is great data - indeed, 8 packages seems eminently fixable.

> So yeah, with the exception of dash, this looks fairly good. Let me also
> dive into dash. Unlike the majority of diverters, it diverts in postinst
> rather than preinst to allow controlling /bin/sh via debconf. A similar
> technique is in effect by gpr. In any case, this is special, because
> dash diverts its own files, so when moving dash's file, its diversions
> can be migrated at the same time. It merely means, that we cannot have
> debhelper just move files (as that would horribly break dash) and
> instead have to move files on a package-by-package way. We could also
> opt for removing dash's diversion in the default case and there even is
> a patch for doing so (#989632) since almost two years. Too bad we didn't
> apply it. In any case, as long as the file moving is not forced via
> debhelper, dash should be harmless.

If I understand correctly, by "forced via debhelper" you mean the
proposal of fixing the paths at build time, right? But if not via
that, it means having to fix all of them by hand, which is a lot - is
it possible to fix dash instead? Or else, we could add an opt-out via
one of the usual dh mechanisms, and use it only in dash perhaps?

> With this number, another option is on the table. Rather than divert
> dpkg-divert, we could just fix these 8 packages to duplicate their
> diversions for /usr and then when moving the underlying files add
> versioned Conflicts to the old version of diverters (none of which are
> essential). So this is an order of 15 uploads (8 diverters, 6 diverted
> packages, dash).
>
> Luca Boccassi kindly pointed me at config-package-dev though. This is a
> tool for generating local packages and it also employs dpkg-divert.
> There is a significant risk of breaking this use case. If we were to
> divert dpkg-divert and automatically duplicate diversions, this use case
> were automatically covered.
>
> I am unsure how to proceed here and request assistance from the
> debathena project to evaluate the situation. If possible, I'd like to
> avoid the complexity of wrapping dpkg-divert.

Which part of config-package-dev causes a conflict here? Is it
something that can be fixed? Given it's declarative, an upload + rdeps
rebuild should be all that's needed, assuming we know what the issue
is and how to fix it. As far as I can remember, it's a build-time
utility and everything it does is embedded in the target package's
maintainer scripts. But it's been a few years since I last used it, so
I might remember wrongly.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Fri, 28 Apr 2023 at 10:12, Luca Boccassi  wrote:
>
> On Fri, 28 Apr 2023 at 09:09, Helmut Grohne  wrote:
> > So yeah, with the exception of dash, this looks fairly good. Let me also
> > dive into dash. Unlike the majority of diverters, it diverts in postinst
> > rather than preinst to allow controlling /bin/sh via debconf. A similar
> > technique is in effect by gpr. In any case, this is special, because
> > dash diverts its own files, so when moving dash's file, its diversions
> > can be migrated at the same time. It merely means, that we cannot have
> > debhelper just move files (as that would horribly break dash) and
> > instead have to move files on a package-by-package way. We could also
> > opt for removing dash's diversion in the default case and there even is
> > a patch for doing so (#989632) since almost two years. Too bad we didn't
> > apply it. In any case, as long as the file moving is not forced via
> > debhelper, dash should be harmless.
>
> If I understand correctly, by "forced via debhelper" you mean the
> proposal of fixing the paths at build time, right? But if not via
> that, it means having to fix all of them by hand, which is a lot - is
> it possible to fix dash instead? Or else, we could add an opt-out via
> one of the usual dh mechanisms, and use it only in dash perhaps?

Also as pointed out by the maintainer the debconf machinery was
dropped from dash, last year:
https://salsa.debian.org/debian/dash/-/commit/c322a1c9fc6be11d7eb4439

This should hopefully simplify things?

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Fri, 5 May 2023 at 17:38, Andreas Metzler  wrote:
>
> On 2023-05-05 Simon Richter  wrote:
> [...]
> > My proposal would be to put the onus on the client registering the
> > diversion:
> [...]
> >  - packages are encouraged to register both diversions
>
> Hello,
>
> That seems to be a rather ugly user interface, ("There is dpkg-divert on
> Debian, but because the usrmerge you need to invoke it twce to be
> sure"). Will we need to have a meta-transition years from now trying to
> get get rid of the double diversions?

It is not the prettiest thing but it is a very clever solution.
Perhaps it could be mitigated with an addendum that makes it optional,
and to be used only when strictly needed, after all moving files
within the same package is fine, it's only the combination of moving
location _and_ package that causes problems. In other words:

- every package is forcefully canonicalized as soon as trixie is open
for business
- the moratorium on moving files from bin/ sbin/ lib/ _and_ to other
packages at the same time is maintained from bookworm till trixie, and
will lifted after trixie ships, and applies implicitly to all the
~2000 binary pkgs that are affected by the above
- the moratorium can be bypassed by a maintainer if and only if the
appropriate conflicts/replaces/diverts as discussed in this thread are
put in place and left there for as long as needed (TBD whether this
means for trixie's cycle or for trixie+1)

In practice, I suspect that out of ~2000 packages shipping bin/ sbin/
lib*/, only a small fraction would end up needing to further move
files out to other packages, so the divert dance requirement can be
restricted only to those. This way impact is minimized, required
testing is smaller, and we get in the final state on day one of trixie
dev cycle.
Moving files between packages already requires busywork anyway, so a
bit more won't hurt that much, especially if we figure out a way to
provide the functionality with a dh addon or such to do the heavy
lifting.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 6 May 2023 at 06:21, Simon Richter  wrote:
>
> Hi,
>
> On 06.05.23 07:11, Luca Boccassi wrote:
>
> > - every package is forcefully canonicalized as soon as trixie is open
> > for business
>
> You will also need to ship at least
>
>   - /lib -> usr/lib (on 32 bit)
>   - /lib64 -> usr/lib64 (on 64 bit)
>
> as a symlink either in the libc-bin package or any other Essential
> package, to fulfill the requirement that unpacked Essential packages are
> operational.

Sure, that doesn't sound problematic? We'll need the same for bin/ and
sbin/ for at least a cycle as you already pointed out. Sounds like a
job for base-files.

In the far future I'd like for these details to be owned by image
builders/launchers/setup processes rather than a package, but this can
be discussed separately and independently, no need to be tied to this
effort.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 6 May 2023 at 11:00, Simon McVittie  wrote:
>
> On Fri, 05 May 2023 at 23:11:54 +0100, Luca Boccassi wrote:
> > In practice, I suspect that out of ~2000 packages shipping bin/ sbin/
> > lib*/, only a small fraction would end up needing to further move
> > files out to other packages
>
> I think the most common case for this is likely to be systemd system
> units, which are currently in /lib/systemd and are sometimes moved between
> binary packages. Splitting out dbus-system-bus-common during the bookworm
> release cycle is a recent example, but it also seems reasonably common
> to move systemd units around as part of having a distinction between
> "the ready-made system service" and "the binary you can run by hand"
> (see apache2 vs. apache2-bin).
>
> udev rules are in a similar situation: consumed by an important system
> service, but shipped by any random package that wants to adjust its
> behaviour.
>
> Two things about systemd units make them a relatively difficult case
> for distro-wide changes like this:
>
> For /bin, /sbin and to a lesser extent /lib/TUPLE, we can often assume
> that only "core" packages (whose maintainers should be paying attention to
> subtleties like this) are affected by any change, because the historical
> definition of those directories was "just enough to boot the system or
> do disaster recovery", minimizing what should be there; but the number
> of packages that touch /lib/systemd is rather large, and a lot of them
> are leaf or near-leaf packages.
>
> Also, they're managed (and sometimes installed) by debhelper, which
> needs to be able to do the right thing relatively automatically. For
> example, if maintainers need to be able to take some special action at
> the point at which their /lib/systemd units move to /usr/lib/systemd,
> then I think debhelper installing into /usr/lib/systemd would have to
> be gated by a compat level change.

Yes, I suspect you are right that it would be the bulk of the leaf
packages. But still shouldn't invalidate what I mentioned? As long as
they are kept in the same package, there shouldn't be a need for
diversions. And if they need to be moved in this cycle, the diversion
dance will be needed. Do you see any problems with this approach?

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 6 May 2023 at 16:11, Simon Richter  wrote:
>
> Hi,
>
> On 06.05.23 21:28, Luca Boccassi wrote:
>
> [shipping usrmerge symlinks in packages]
>
> > In the far future I'd like for these details to be owned by image
> > builders/launchers/setup processes rather than a package, but this can
> > be discussed separately and independently, no need to be tied to this
> > effort.
>
> Ideally I'd like to have this information in a single package rather
> than distributed over ten different tools, especially as this is also
> release and platform dependent.
>
> If possible, I'd like to go back to the gold standard of
>   - download Essential: yes packages and their dependencies
>   - unpack them using dpkg --fsys-tarfile | tar -x
>   - install over this directory with dpkg --root=... -i *.deb
>
> to get something that works as a container. Right now, that only works
> if I remove "init" from the package list, which is fair since a
> container doesn't need an init system anyway.
>
> The less an image builder needs to deviate from this approach, the
> better for our users.

To have a working system you need several more steps that are
performed by the instantiator/image builder, such as providing working
and populated proc/sys/dev, writable tmp/var, possibly etc. And it
needs to be instantiated with user/password/ssh certs/locale/timezone.
And if it needs to be bootable on baremetal/vm, it needs an ESP. And
then if you have an ESP and want to run in a VM with SB, you'll need
self-enrolling certs on first use or ensuring the 3rd party CA is
provisioned. And then...

You get the point. Going from a bunch of packages to a running system
necessarily has many steps in between, some that are already done and
taken for granted, for example when you say "works as a container" I'm
pretty sure the "container" engine is taking care of at the very least
proc/dev/sys for you, and it's just expected to work. bin -> usr/bin,
sbin -> usr/sbin and lib -> usr/lib should get the same treatment: if
they are not there, the invoked engine should prepare them. systemd
and nspawn have been able to do this for a while now.

Not having those hard coded means that the use case of / on a tmpfs
with the rest instantiated on the fly, assembled with the vendor's
/usr and /etc trees, becomes possible, which is neat. And said trees
can pass the checksum/full integrity muster.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 6 May 2023 at 19:51, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Sat, May 06, 2023 at 04:52:30PM +0100, Luca Boccassi wrote:
> > To have a working system you need several more steps that are
> > performed by the instantiator/image builder, such as providing working
> > and populated proc/sys/dev, writable tmp/var, possibly etc. And it
> > needs to be instantiated with user/password/ssh certs/locale/timezone.
> > And if it needs to be bootable on baremetal/vm, it needs an ESP. And
> > then if you have an ESP and want to run in a VM with SB, you'll need
> > self-enrolling certs on first use or ensuring the 3rd party CA is
> > provisioned. And then...
>
> You paint it this way, but it really used to just work until we got the
> /usr-merge. Indeed, debvm creates virtual machine images effectively by
> bootstrapping a filesystem from packages and turning the resulting tree
> into a file system image.
>
>  * /proc, /sys, /dev are mounted by systemd. All you need to do here is
>create the directories and base-files does so.
>  * /tmp is shipped by base-files.
>  * user and password creation is not handled yet, but can be handled by
>something similar to systemd-firstboot.
>  * Not sure what you mean with certs, locale and timezone. You can just
>install ca-certificates, locales and tzdata as part of the bootstrap.
>  * The bootloader part for baremetal is kinda out of scope for
>bootstrap, which is why debvm side-steps this. You can also skip it
>for containers and build chroots. So it is one out of multiple use
>cases that needs extra work here.
>
> In a good chunk of situations, you can get just by without messing
> around. Well that is until we broke it via usr-is-merged. I concur with
> Simon Richter, that restoring this property is a primary concern.
>
> > You get the point. Going from a bunch of packages to a running system
> > necessarily has many steps in between, some that are already done and
> > taken for granted, for example when you say "works as a container" I'm
> > pretty sure the "container" engine is taking care of at the very least
> > proc/dev/sys for you, and it's just expected to work. bin -> usr/bin,
> > sbin -> usr/sbin and lib -> usr/lib should get the same treatment: if
> > they are not there, the invoked engine should prepare them. systemd
> > and nspawn have been able to do this for a while now.
>
> No, this misses the point. You can configure essential in a very limited
> environment. However, you cannot do so without the lib or lib64 symlink
> (depending on the architecture) and the bin symlink. This is so
> critical, that it cannot be deferred to some external entity. It must be
> part of the bootstrap protocol. There are some suggested ways to fix
> this (such as adding separate bootstrap scripts next to maintainer
> scripts), but nothing implemented.
>
> > Not having those hard coded means that the use case of / on a tmpfs
> > with the rest instantiated on the fly, assembled with the vendor's
> > /usr and /etc trees, becomes possible, which is neat. And said trees
> > can pass the checksum/full integrity muster.
>
> It's neat that you can solve your use case by breaking other people's
> use cases. This is not constructive interaction however. This kind of
> behaviour is precisely what caused so much conflict around the
> /usr-merge. What if I gave a shit for your use case? Denying the
> /usr-merge and just continuing unmerged as long as possible (as merging
> would break my use case) would be my strategy of choice. You can make a
> difference here by starting to recognize other people's use cases and
> proposing solutions in that merged world. And no, it's not "add duct
> tape to every bootstrap tool".
>
> So I really want to see a solution for the bootstrap protocol before
> moving the dynamic linker and /bin/sh to its canonical location. The
> current bootstrap protocol is kept on life-support by installing the
> usrmerge package by default. Dropping usrmerge from the
> init-system-helpers dependency as first alternative or moving the
> dynamic linker would break it. If I had a solution in mind, I'd
> definitely post it right here, but unfortunately I have not.
>
> Helmut

There is absolutely no need to be so aggressive. I care about both use
cases, but evidently you have formed a different view in your mind, so
nothing productive is going to come out of this subthread right now,
so I'll stop replying. It's off-topic anyway.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 6 May 2023 at 19:51, Helmut Grohne  wrote:
>
> Hi Luca,
>
> I fear you are still missing too many relevant details. What sounds like
> a simple plan is severely broken if carried out in this simple form. It
> disappoints me that you continue to present it as this simple given the
> earlier evidence.
>
> On Fri, May 05, 2023 at 11:11:54PM +0100, Luca Boccassi wrote:
> > - every package is forcefully canonicalized as soon as trixie is open
> > for business
>
> I gave you scripts to prototype this (via binary patching) and
> demonstrated that this immediately makes a fair number of packages rc
> buggy due to causing unpack errors. I used zutils as an example. It
> really is not that simple and it absolutely cannot be done forcefully,
> but must be done in an opt-in way. I won't repeat the rationale here.

Sure, there are some things that need special handling, as you have
pointed out. What I meant is that I don't think we need special
handling for _all_ affected packages. AFAIK nothing is using
diversions for unit files or udev rules, for example (I mean if any
package is, please point it out, because I would like a word...). I
very strongly suspect this will be a small minority out of the total
of ~2k would need this treatment, and the vast majority would not. Do
you disagree with this gut feeling?
Of course, it goes without saying, we should check this before going
forward in any direction.

> > - the moratorium on moving files from bin/ sbin/ lib/ _and_ to other
> > packages at the same time is maintained from bookworm till trixie, and
> > will lifted after trixie ships, and applies implicitly to all the
> > ~2000 binary pkgs that are affected by the above
>
> While the CTTE placed the moratorium until the release of bookworm, the
> release team extended it beyond, see
> https://lists.debian.org/e1ocdqk-0005ge...@respighi.debian.org. So you
> need explicit agreement from the release team on your plan. Failing
> that, any package that has been forcefully moved is immediately
> rc-buggy due to failing a release team requirement.

Of course the release team needs to be on board, no questions about
that. But given the idea is to maintain their decision exactly as it
stands I wouldn't imagine it would be an issue? Once again, the
moratorium is explicitly about moving between locations _and_
packages, in combination, not either/or. From that same email you
linked:

"Files moving their canonical location between / and /usr (details in
[1]) *and* from one binary package to another binary package within
one release cycle remain an RC issue unless dpkg supports it."

I'm proposing to keep this in place as a general rule, with the new
escape hatch that you devised as the only addition.

> Again, let me try to fix up your plan:
>
> 1. Implement a service that reliably notices missing Breaks + Replaces
>even in the presence of aliasing. Augment it to notice
>canonicalization changes and have it report that Conflicts (or other
>measures) are needed in such cases. This should operate for upgrades
>from stable to testing, stable to unstable and testing to unstable.

There are already distro-wide upgrade piuparts checks run occasionally
IIRC, at least I've seen a bug from one being reported this week, so
we should be most of the way there already?
To be clear, this would be very nice and welcome to have obviously,
but I don't think it needs to be a blocker. We don't have such checks
for vast parts of Policy, including moving files without
Breaks+Replaces as evidenced by the recent MBF, and yet we managed to
survive thus far. I don't think it's fair that this workstream should
be held to higher standards than the rest of the project.

> 2. Add a usr-merge-helper script to usr-is-merged that helps with
>instating temporary diversions for the purpose of avoiding file loss.
>
> 3. Add a dpkg-divert wrapper for duplicating diversions according to
>aliasing to usr-is-merged. Add postinst scripts that duplicate
>existing diversions.

Yep, sounds good

> 4. Add canonicalization support to debhelper as an (internal) addon.
>Enable this addon in the next compat level. This will again populate
>${misc:Pre-Depends} with "usr-is-merged (>= fixed version)" as needed.
>Note that usrmerge's provides is unversioned and doesn't satisfy such
>a dependency.

As already mentioned, I do not believe this is necessary for _all_
cases. It is necessary for a certain number (that we should ascertain
beforehand!) of cases, and we need the machinery implemented for them,
but I don't think we should impose this workflow with pre-depends and
diversions for all affected packages. I think it should be mandatory
for problematic packages such as those you already pointed

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sun, 7 May 2023 at 10:14, Ansgar  wrote:
>
> Hi,
>
> On Sun, 2023-05-07 at 07:50 +0200, Helmut Grohne wrote:
> > But then, you only capture diversions inside Debian's ecosystem
>
> It's unreasonable to support stuff outside Debian's ecosystem: even
> basic dependency relations do not work for this.
>
> Debian's dependency system requires to explicitly declare
> Depends/Conflicts/Replaces/Breaks, but for obvious reasons we cannot do
> that for packages outside Debian's ecosystem.
>
> The same is true for diversions/alternatives/* or anything else
> requiring coordination among all users: the dpkg ecosystem has too many
> practical limitations to support non-Debian packages on anything but a
> "it might work" basis (which is usually "good enough").  (This is even
> true for packages within the Debian ecosystem, especially when one
> considers partially implemented features like multi-arch.)
>
> Is there any specific reason why specifically diversions are a problem
> where "it might work" is not sufficient? That is, why should we divert
> from the usual standard for dealing with packages outside the Debian
> ecosystem here?
>
> > I also caution that we've started from a very simple approach and tried
> > fixing it up to address the problems that we recognized in analyzing it.
> > My impression is that we are not finished with our discovery here and
> > won't be for quite some time.
>
> Well, we find limitations in dpkg that we in all other contexts usually
> ignore. If we used similar expectations in other cases, we would need
> to very much restrict when Breaks/Conflicts/Replaces might be used at
> all: it's totally unrealistic to list all (possibly local) packages
> that ship conflicting files, possibly only created by maintainer
> scripts. Or to explicitly list all reverse dependencies that might be
> broken by a particular change. We also would not have multi-arch yet as
> the dependency system doesn't support it fully (some of which is
> already known, but probably discovery isn't finished yet).
>
> (Of course in some cases explicitly listing reverse dependencies can be
> avoided: just always introduce something like
>
> Provides: ${foo}-compat (= 1)
>
> for *all* dependencies and forbid `>=` in `Depends`; this allows to
> stop providing that in cases where one would have to declare explicit
> `Breaks` before. But only the direct provider can use this, so it's
> already too limited... Alternatively forbid *all* changes that would
> require this, i.e., require stable interfaces. However we do not do
> this.)
>
> But for all these issues we just say "meh, you are out of luck".

Indeed, if we don't worry about local random changes or random third
party packages beyond documenting what needs attention, we shouldn't
worry about them for this either. As already mentioned lots of third
party packages don't even use Debian's toolchain to build packages, so
there's nothing that we can do anyway.

Kind regards, Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sun, 7 May 2023 at 00:44, Sean Whitton  wrote:
>
> Hello,
>
> On Sat 06 May 2023 at 09:47PM +01, Luca Boccassi wrote:
>
> > On Sat, 6 May 2023 at 19:51, Helmut Grohne  wrote:
> >>
> >> > - the moratorium on moving files from bin/ sbin/ lib/ _and_ to other
> >> > packages at the same time is maintained from bookworm till trixie, and
> >> > will lifted after trixie ships, and applies implicitly to all the
> >> > ~2000 binary pkgs that are affected by the above
> >>
> >> While the CTTE placed the moratorium until the release of bookworm, the
> >> release team extended it beyond, see
> >> https://lists.debian.org/e1ocdqk-0005ge...@respighi.debian.org. So you
> >> need explicit agreement from the release team on your plan. Failing
> >> that, any package that has been forcefully moved is immediately
> >> rc-buggy due to failing a release team requirement.
> >
> > Of course the release team needs to be on board, no questions about
> > that. But given the idea is to maintain their decision exactly as it
> > stands I wouldn't imagine it would be an issue? Once again, the
> > moratorium is explicitly about moving between locations _and_
> > packages, in combination, not either/or. From that same email you
> > linked:
> >
> > "Files moving their canonical location between / and /usr (details in
> > [1]) *and* from one binary package to another binary package within
> > one release cycle remain an RC issue unless dpkg supports it."
> >
> > I'm proposing to keep this in place as a general rule, with the new
> > escape hatch that you devised as the only addition.
>
> Actually, the morotorium is not explicitly only about moving between
> both locations and packages.  Firstly, the TC's morotorium does not have
> the qualification, restricting *any* movements in data.tar.*:
>
> The Technical Committee recommends that during the Debian 12
> development cycle, the maintainers of individual packages should not
> proactively move files from the root filesystem to the corresponding
> locations in /usr in the data.tar.* of packages. Files that were in
> /usr in the Debian 11 release should remain in /usr, while files
> that were in /bin, /lib* or /sbin in the Debian 11 release should
> remain in those directories.  If files were moved from /bin, /lib*
> or /sbin into /usr since the Debian 11 release, they should be moved
> back to their Debian 11 locations.
>
> Then secondly, the RT's message is ambiguous, because it says both that
> they want the /TC's/ morotorium to remain in place, and also that files
> moving between both locations and packages is an RC issue.
>
> Until the RT's position is clarified, I think we should treat the
> broader prohibition as what they require.  The TC are going to discuss
> this issue at our meeting on Tuesday, and one possible outcome is that
> we reissue our version of the broader prohibition.

Sure, this is in the context of the ongoing discussion in the TC about
revising their side of the advice. Also, we shouldn't lose sight of
the reason why this was issued in the first place: it is designed to
stop a problem from happening, and that problem can only happen when
both conditions are true. I can't read minds obviously, but I imagine
that's the reason the RT advice was worded as it was.

> Stepping back:
>
> I am far from being an expert on the details of merged-/usr.  But one
> thing I've noticed is that among the people who have spent the most time
> looking into it, the majority think that simple fixes are not going to
> be sufficient.  Only a few people who have spent a lot of time on it
> still think that the fixes that are required are relatively simple ones.
>
> If the former group are wrong then the transition takes longer than it
> needs to, but we don't lose any confidence in the state of our users'
> installations.
>
> If the latter group are wrong then we'll probably end up with a longer
> transition anyway, and a worse situation for both our maintainers and
> our users.  And it's within one of the areas of Debian that we're most
> proud of -- completely smooth upgrades between stable releases.
>
> So, I think we should assume the people who are more worried are right,
> for the time being.  We lose little in doing so.

Pondering about things is all well and good, however, I think the
'worry' needs to be realistic (eg: a conflict due to /bin/foo and
/usr/bin/foo both existing is not something that needs to cause any
worrying, as it is already disallowed by policy and unsupported), and
congruent with usual practices that are applied for all other changes
and processes. IE, the appropriate weighting should be given to
potential problems spotted.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sun, 7 May 2023 at 06:55, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Sat, May 06, 2023 at 09:47:15PM +0100, Luca Boccassi wrote:
> > Sure, there are some things that need special handling, as you have
> > pointed out. What I meant is that I don't think we need special
> > handling for _all_ affected packages. AFAIK nothing is using
> > diversions for unit files or udev rules, for example (I mean if any
> > package is, please point it out, because I would like a word...). I
>
> I've posted a list in
> https://lists.debian.org/20230428080516.ga203...@subdivi.de and indeed,
> udev rules are being diverted in one case.

*fetching sledgehammer*

> But then, you only capture diversions inside Debian's ecosystem and miss
> out on other kinds of diversions such as local diversions. We currently
> support imposing local diversions on pretty much arbitrary files
> including unit files. And while we've occasionally moved files between /
> and /usr before the transition, doing it for 2000 packages significantly
> grows the risk of it affecting someone. So really, we want all such
> diversions duplicated before unpacking a package the has moved its
> files. The way to achieve that is Pre-Depending on usr-is-merged. To me,
> this sounds like we really want some special handling for all affected
> packages.

The local/external aspect is already covered in Ansgar's reply and subthread.

> I also caution that we've started from a very simple approach and tried
> fixing it up to address the problems that we recognized in analyzing it.
> My impression is that we are not finished with our discovery here and
> won't be for quite some time. This uncertainty of what else we might
> break is the most significant downside I see with your approach.

Sure, but adding changes that are (seemingly) unnecessary for a large
percentage of affected packages also brings uncertainty. Every
software has bugs, thus it follows that injecting more software in the
way of a package being installed will likely also inject bugs. Which
doesn't mean we shouldn't consider it, however, it should be weighted
appropriately.

> > very strongly suspect this will be a small minority out of the total
> > of ~2k would need this treatment, and the vast majority would not. Do
> > you disagree with this gut feeling?
>
> I do disagree indeed. While the special handling may be mostly
> mechanical for the majority of packages, it still see it as required.
>
> Worse, we also need to discuss how this affects backporting of packages.
> Any package enabling the addon needs to have the addon removed for a
> backport to undo the move. Worse, when backporting debhelper, any
> package that uses the new compat level must explicitly disable the
> addon. And then we may need to fix upgrade paths from backports to
> stable.

Packages that need special handling will need special handling for
backporting too. This is nothing new, there was never a project-wide
guarantee that a package uploaded to testing can apply 1:1 to
backports, it is common enough to require changes/reverts/adjustments,
and if it's fine to require that in other cases, it's fine for this
case too.
If the majority of packages are simply converted, with no manual
handling and no diversion, then it should be simple to handle: the
debhelper in stable will not perform the conversion by definition as
the logic won't be present, and any dh upload to backports will have
such logic disabled, so that other packages that get uploaded to
backports and built with either the stable or the backports debhelper
won't have any change performed on them.

> > Curve ball question: is there anything blocking us from canonicalizing
> > PT_INTERP at the source in gcc so that the essential set (and
> > everything else, really) can work without the lib -> usr/lib symlink?
> > Would anything obvious break?
> > This would be in trixie, so even when unpacked on bookworm for the
> > upgrade case, the loader is guaranteed to be accessible from the
> > canonical path. Heck, we could even consider canonicalizing shebangs
> > in scripts shipped in essential packages? I'd imagine /bin/sh would
> > have the same issue as the loader?
>
> Heh! As much as this initially sounds insanely crazy,

I know, right? And yet...

> I was unable to
> spot a reason for why this cannot work at all. It definitely sounds like
> fixing the bootstrap protocol to me. On the flip side, I'd assume the
> loader location (for the purpose of examining ELF objects) to be hard
> coded in quite some places.  Changing gcc seems out of question, because
> we also expect binaries built on Debian to run on non-Debian systems.

I don't think this is true? At least not in t

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sun, 7 May 2023 at 12:51, Luca Boccassi  wrote:
>
> On Sun, 7 May 2023 at 06:55, Helmut Grohne  wrote:
> >
> > Hi Luca,
> >
> > On Sat, May 06, 2023 at 09:47:15PM +0100, Luca Boccassi wrote:
> > > Sure, there are some things that need special handling, as you have
> > > pointed out. What I meant is that I don't think we need special
> > > handling for _all_ affected packages. AFAIK nothing is using
> > > diversions for unit files or udev rules, for example (I mean if any
> > > package is, please point it out, because I would like a word...). I
> >
> > I've posted a list in
> > https://lists.debian.org/20230428080516.ga203...@subdivi.de and indeed,
> > udev rules are being diverted in one case.
>
> *fetching sledgehammer*

Filed https://bugs.debian.org/1035667 with extreme prejudice, and MR
up to fix it too.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sun, 7 May 2023 at 22:10, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Sun, May 07, 2023 at 12:51:21PM +0100, Luca Boccassi wrote:
> > The local/external aspect is already covered in Ansgar's reply and 
> > subthread.
>
> I hope that we can at least agree that we don't have consensus on this
> view. And the more I think about it, the more it becomes clear to me
> that this non-consensus is part of the larger disagreement we have about
> this whole transition. Do you see any way towards getting to common
> ground here?

I can see we don't agree on this matter, of course, that is clear. And
I hope we can find common ground. But let me provocatively ask this
first: is the same rule going to be enforced for all other changes
that happen in the project that might affect external packages? If
anybody points out past changes, recent or less recent, that caused
issues for third party packages, will the TC ask for those changes to
be reverted or otherwise modified accordingly? Will a change to Policy
be proposed that spells out that third party packages cannot ever be
broken, no matter what they do, and must always work?

> > Sure, but adding changes that are (seemingly) unnecessary for a large
> > percentage of affected packages also brings uncertainty. Every
> > software has bugs, thus it follows that injecting more software in the
> > way of a package being installed will likely also inject bugs. Which
> > doesn't mean we shouldn't consider it, however, it should be weighted
> > appropriately.
>
> Let me put this into perspective. In this scenario, we will have a few
> packages with versioned Pre-Depends on usr-is-merged. The seemingly
> unnecessary change here is adding more Pre-Depends of the same kind to
> many more packages. It seems very likely to me that one of the few
> Pre-Depends will cause usr-is-merged to be upgraded early and thus those
> possibly unnecessary Pre-Dependencies will be harmless. Do you actually
> have some scenario in mind that would warrant judging this as risky
> beyond suspicion? (Which is not to say that there is no risk as the
> whole affair bears quite some risk.)

The more pre-depends, the more constraints we put on apt. I do not
have a specific scenario in mind as we don't even have a full set of
changes to look at, but it seems clear to me it will have _some_
effect, no?

> > Packages that need special handling will need special handling for
> > backporting too. This is nothing new, there was never a project-wide
> > guarantee that a package uploaded to testing can apply 1:1 to
> > backports, it is common enough to require changes/reverts/adjustments,
> > and if it's fine to require that in other cases, it's fine for this
> > case too.
>
> It seems that you missed my argument and it likely wasn't spelled out
> explicitly enough, so let me retry. Yes, you may need to adapt packages
> that are being backported. We don't disagree about that (and hope people
> get it right, which they won't, but so be it). The really bad thing here
> is that a backports upload may require changes to the package in
> unstable!
>
> Say we packaged foo version 1 in stable and it puts everything in /bin.
> Then we update foo to version 2 in unstable and foo gains a new
> /bin/bar. Due to the debhelper addon, this is actually shipped as
> /usr/bin/bar. Great. Then we backport foo version 2 to stable. Given
> that debhelper no longer moves, it'll be /bin/bar. Then we notice that
> foo is not laid out nicely and we split a bar package from it in version
> 3 and move /usr/bin/bar into bar. Now a user may install stable, install
> foo version 1, install the foo version 2 backport and then update to
> nextstable. In that stable upgrade, bar version 3 may be unpacked before
> foo version 3 and as a result /usr/bin/bar goes missing when the
> backported foo version 2 gets upgraded to the regular foo version 3 as
> this deletes /bin/bar.
>
> So when we backport a package, the unstable package may need to be
> modified to avoid such unpack file loss scenarios. In a simple case, we
> may be able to just add Conflicts, but the takeaway is that backporting
> a package may now break upgrades to nextstable in a way that requires
> fixes in nextstable to accommodate for such upgrades.

Sure that's a legitimate concern, however, wouldn't it fall into the
"needs special handling" bucket? It is a case where the file is moving
both in location and package, so it is covered by the blank statement
"either don't do that or implement the required workaround via
diversion/conflict/etc". What am I missing?

> > If the majority of packages are simply converted, with no manual
> > handling and no 

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Mon, 8 May 2023 at 03:57, Simon Richter  wrote:
>
> Hi,
>
> On 5/7/23 18:14, Ansgar wrote:
>
> > Is there any specific reason why specifically diversions are a problem
> > where "it might work" is not sufficient? That is, why should we divert
> > from the usual standard for dealing with packages outside the Debian
> > ecosystem here?
>
> Locally created diversions are a supported feature, and the only way for
> admins to replace single files in a way that is safe for installing updates.
>
> Even within Debian, it is not sufficient to just coordinate uploads of
> packages that divert each others' files, because the new diversion needs
> to be in place before a newly-canonicalized package is unpacked, a
> Breaks relationship does not enforce that ordering, and while a
> Conflicts without a Replaces does, this adds a lot of constraints to the
> solver.

Sure, they are supported in the sense that they can be enabled, and
then you get to keep the pieces. We ship thousands of maintainer
scripts, and I have never seen one that takes into account completely
arbitrary and random possible local diversion, apart from dash for
/bin/sh (and we are about to nuke most of it!), when
moving/adjusting/fixing and whatnot. Do you have any such
counter-example in mind?

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

Will get to the rest later tonight, two quick points:

On Mon, 8 May 2023 at 09:58, Helmut Grohne  wrote:
> > But the more I think about it, the more I am convinced that the
> > default option working best for Debian is the one that matches the
> > project's choice of a filesystem layout. After all, this is
> > configurable in the toolchain for a reason.
> > And the vast majority of the rest of the world has long since finished
> > this transition, so I struggle to think where software built with this
> > default wouldn't work. Bullseye will be oldoldstable at that point,
> > and even that was default merged for new installations, and really old
> > ones (oldoldoldoldstable at that point? I lost count) will be long
> > EOL. I suppose they could still be around unmaintained, but who uses a
> > toolchain from 8 years in the future to build software for an EOL
> > distribution 8 years in the past? Normally it's the other way around,
> > as even glibc adds new symbols and is not forward compatible.
>
> This seems somewhat convincing to me. Would you reach out to toolchain
> maintainers to discuss this as an early change after the release of
> bookworm?

Have done so now via the gcc mailing list.

> > On the ELF interpreter, as long as we can reasonably ensure it works,
> > I do believe we should switch it, regardless of what we do with the
> > symlinks, how we ship/add/build/package/create/manage them, as a
> > desired final state. Again, we should make the default in Debian work
> > for Debian. And given the default for Debian from Bookworm onward is
> > that the loader is in /usr/lib/, it seems perfectly reasonable to me
> > that it software built for Debian and shipped in Debian should look
> > there for it.
>
> I suppose that we've been confusing the different approaches here. The
> question of what links base-files should contain mostly arises if you
> start from the assumption that we do not modify the ELF interpreter
> location. Once changing its (and /bin/sh's) location, the question of
> how to install those symlinks can indeed be done in base-files.postinst
> or at some other place where dpkg doesn't have to know much about it
> indeed. Would you agree to examine the approach where we don't modify
> the ELF interpreter location in parallel as a backup plan?

Yeah we definitely should do that. I think we should separate a bit
long-term vs short-term on that front, as it will help reach a
conclusion more quickly. I think that aspect is easy to revise, and
shouldn't lock us in a particular position one way or the other.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Mon, 8 May 2023 at 09:58, Helmut Grohne  wrote:
>
> On Mon, May 08, 2023 at 02:07:08AM +0100, Luca Boccassi wrote:
> > I can see we don't agree on this matter, of course, that is clear. And
> > I hope we can find common ground. But let me provocatively ask this
> > first: is the same rule going to be enforced for all other changes
> > that happen in the project that might affect external packages? If
> > anybody points out past changes, recent or less recent, that caused
> > issues for third party packages, will the TC ask for those changes to
> > be reverted or otherwise modified accordingly? Will a change to Policy
> > be proposed that spells out that third party packages cannot ever be
> > broken, no matter what they do, and must always work?
>
> I'm not sure about the TC's role in this. For the record, I am doing all
> of the analysis (and design work) in this thread without a TC hat. I
> also cannot comment on what the TC is going to rule this matter. Can we
> leave that aside or formally file it there if you see a need?

Sure, it was just to keep it impersonal. Substitute TC with
"interested party" or so.

> I agree that what we support is vague at best and we can readily see
> from earlier conflicts that this is a recurring matter. We still
> disagree over how much maintainers should support sysvinit. I've also
> quite recently failed at properly preparing a transition (non-essential
> adduser) and while we could write about it in release-notes, what is
> going to happen is that we'll revert it for bookworm and then I can
> retry properly.
>
> You may also have noticed that my analysis of possible problems in this
> thread very much reasons about packages shipped in Debian releases. I
> would actually like to call external packages and local diversions
> unsupported, but I was rightfully criticised that this is falling short.
>
> So no, I cannot tell you where the boundary of our support is. I
> initially assumed it to be closer to where you paint it and am now
> trying to adapt to meet the expectations of others.
>
> For instance, I've also reached out to DSA and inquired on their use.
> While I haven't found local diversions or local statoverrides in
> dsa-puppet.git, it seems that a number of external packages ship files
> in /sbin or /lib (including udev rules and systemd units).

I think the question of what do we want to support is an important
one, and I care greatly that for this particular endeavour we do not
impose a higher burden on ourselves that would otherwise be expected
in different situations. If expectations are shifting considerably, we
should codify it, so that everyone is held up to the same standards.

> > The more pre-depends, the more constraints we put on apt. I do not
> > have a specific scenario in mind as we don't even have a full set of
> > changes to look at, but it seems clear to me it will have _some_
> > effect, no?
>
> We've been there with multiarch-support and my experience with that
> suggests that the primary effect is increasing the size of Packages
> files. Though given that you are obviously worried here, I suppose more
> research is warranted.

If that's the only thing to worry about, then I'm not worried at all!

> > Sure that's a legitimate concern, however, wouldn't it fall into the
> > "needs special handling" bucket? It is a case where the file is moving
> > both in location and package, so it is covered by the blank statement
> > "either don't do that or implement the required workaround via
> > diversion/conflict/etc". What am I missing?
>
> You are missing the distribution of responsibility. Quite commonly,
> backports are performed by someone else than the package maintainer.
> Yet, an uncoordinated backport can now render the package in unstable
> rc-buggy.

Well, sure, but once again any backport changes can break in
interesting and novel ways. The simplest of them all is getting the
version wrong so that updates do not work... and yet we have very
little if anything in the way to stop that, no? You can just ignore
Lintian screaming at you and upload...

To me that's just another case to be solved with good documentation
and communication.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Mon, 8 May 2023 at 19:06, Sean Whitton  wrote:
>
> Hello,
>
> On Sun 07 May 2023 at 12:03PM +01, Luca Boccassi wrote:
>
> > Sure, this is in the context of the ongoing discussion in the TC about
> > revising their side of the advice.
>
> I think it's highly unlikely that we revise it rather than just reissue
> it, at the present time, because too many details are unsettled.
>
> > Also, we shouldn't lose sight of the reason why this was issued in the
> > first place: it is designed to stop a problem from happening, and that
> > problem can only happen when both conditions are true. I can't read
> > minds obviously, but I imagine that's the reason the RT advice was
> > worded as it was.
>
> It's designed to stop as-yet-unknown problems happening, too.

Well, sure, but we've been at this for years, any such problems should
really be known by now. This is with Bookworm as it stands of course,
when we go in and make more changes then we obviously have to be
careful, but that's the entire reason this thread exists and is still
going on.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Mon, 8 May 2023 at 21:09, Russ Allbery  wrote:
>
> Sam Hartman  writes:
>
> > As someone who has been following this, I support the work Helmut and
> > Simon Richter have been doing.
>
> > I have more confidence in that view than the one Luca is proposing.
> > I also support Shawn's interpretation that being conservative here is
> > good.
>
> > I think even with my support we have no consensus.  However hopefully we
> > can get a few more people who have been reading the whole thread to
> > chime in and a consensus will appear.
>
> I've also been following this.
>
> I appreciate Luca's questioning of the necessity of parts of the approach
> and looking for simpler solutions; I think that's valuable feedback, and
> we should avoid assuming that every conceivable edge case is supported in
> Debian.  There are unsupported edge cases in Debian already and likely
> always will be because distributions are complex.
>
> That said, I find Helmut and Simon's analysis to be more persuasive so
> far.  I do think we should try to find a fairly robust solution, because
> the feature we're trying to support here (smooth upgrades) is a core and
> defining feature of what makes Debian Debian.  That doesn't mean we need
> to support literally anything someone might have done; that won't be
> possible.  But I think there are going to be enough unanticipated problems
> that we should try to cover the anticipated problems, and that includes at
> least the relatively obvious or known outside-of-Debian uses of things
> like diversions.
>
> I would like to stay open to addressing some of those problems via
> documentation or explicit upgrade instructions where that makes sense.  If
> we have places where there's a choice between writing extremely tricky and
> complicated code versus providing people with simple instructions for how
> to locate problematic diversions on their system, remove them before the
> upgrade, and then put them back afterwards (or accomplish their goal in
> some other way), we should consider taking the documentation approach
> instead.  But that still requires being able to enumerate at least the
> most likely problems and understand them.
>
> For example, if local system administrators have been deactivating systemd
> units by diverting them, at first glance I think it would be better to
> clearly tell them that they should stop doing this and instead use
> masking rather than writing code to try to ensure this continues working.

Obviously agree with your last point, I think documentation is
fundamental for dealing with local changes. When I say that I don't
think we should worry about strange local-only changes I mean exactly
as you said, that I don't think we should start shipping complicated
code that tries to deal with people diverting /sbin/init to
/var/lolcat or whatever. Clear documentation that tells "if you did X
look at Y and do Z" is the bare minimum I'd think.

But again, I do think we need to try and define what it is that we
want to support here. If we are serious about it, then we should
codify it, and hold any future changes to the same standards, wherever
they may come from. If we are not willing to do this, then I have to
ask why.

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Tue, 9 May 2023 at 05:12, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Tue, May 09, 2023 at 01:56:53AM +0100, Luca Boccassi wrote:
> > On Mon, 8 May 2023 at 19:06, Sean Whitton  wrote:
> > > It's designed to stop as-yet-unknown problems happening, too.
> >
> > Well, sure, but we've been at this for years, any such problems should
> > really be known by now. This is with Bookworm as it stands of course,
> > when we go in and make more changes then we obviously have to be
> > careful, but that's the entire reason this thread exists and is still
> > going on.
>
> This actually feels rather worrying to me. On one hand, you say that
> problems should be know. On the other hand, you proposed a simple
> transition with quite a number of problems that you apparently didn't
> see coming. Even relatively simple mechanisms, such as just repacking
> all the .debs to ship files in their canonical location and then trying
> to install them, revealed a dpkg unpack error in zutils. This
> combination of claiming that problems should be known while at the same
> time apparently not knowing them makes me uneasy to move forward here.
>
> So while I want to see the moratorium lifted, it all makes a lot more
> sense to me given what we've seen in this thread. The worst of outcomes
> I see here is the one where we cause problems that don't have a good
> solution as any way forward would break someone's use case (with
> someone's use case often being smooth upgrades in one way or another).
> It's those where we cannot move forward nor revert.

No need to get worried! With "Bookworm as it stands" I mean that
literally. What we ship in Bookworm has been stable for years, so
while something new could be lurking somewhere, it seems vanishingly
unlikely at this stage.

Once we start changing things in Trixie, in whichever way we decide,
that's of course all new and all bets are off.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >
> >The core issue as I see it is as follows:
> >
> >- Debian has decided to support only merged-/usr, including possibly
> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> >  as the interpreter in binaries.
>
> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> seen. The interpreter must *not* be changed willy-nilly.

Nothing's happening 'willy-nilly'. We are discussing a bunch of
seemingly crazy options, as in, "what would _actually_ explode if we
do this or do that?", on this very d-devel thread. I posted a longer
version here some days ago:

https://lists.debian.org/debian-gcc/2023/05/msg00030.html

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 11:40, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
> >On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
> >>
> >> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >> >
> >> >The core issue as I see it is as follows:
> >> >
> >> >- Debian has decided to support only merged-/usr, including possibly
> >> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> >> >  as the interpreter in binaries.
> >>
> >> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> >> seen. The interpreter must *not* be changed willy-nilly.
> >
> >Nothing's happening 'willy-nilly'. We are discussing a bunch of
> >seemingly crazy options, as in, "what would _actually_ explode if we
> >do this or do that?", on this very d-devel thread. I posted a longer
> >version here some days ago:
> >
> >https://lists.debian.org/debian-gcc/2023/05/msg00030.html
>
> Oh holy fuck.
>
> You're talking about changing ABI by doing this. That *is* utterly
> crazy. No.

It's a thought experiment on a mailing list. If we can't even have
those anymore, something went very wrong somewhere.

You seem to be aware of things that wouldn't work anymore (I think?).
If you have a couple of minutes to spare, may I please ask you to
reply to that thread with such examples? I am genuinely interested in
understanding and talking about it. Thank you.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 12:08, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
> >On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
> >>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
> >>>
> >>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >>> >
> >>> >The core issue as I see it is as follows:
> >>> >
> >>> >- Debian has decided to support only merged-/usr, including possibly
> >>> >  moving /bin/sh to /usr/bin/sh or using /usr/lib*/ld-linux-x86-64.so.2
> >>> >  as the interpreter in binaries.
> >>>
> >>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> >>> seen. The interpreter must *not* be changed willy-nilly.
> >>
> >>Nothing's happening 'willy-nilly'. We are discussing a bunch of
> >>seemingly crazy options, as in, "what would _actually_ explode if we
> >>do this or do that?", on this very d-devel thread. I posted a longer
> >>version here some days ago:
> >>
> >>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
> >
> >Oh holy fuck.
> >
> >You're talking about changing ABI by doing this. That *is* utterly
> >crazy. No.
>
> People have asked me to expand on this further...
>
> I've been involved in defining ABI before, specifically for
> armhf. It's not a quick and easy process. It needs buy-in from all
> sides to make things work, and people *really* value the
> interoperability that it enables.
>
> The interpreter path is one of the most important parts of the ABI
> spec, the bit that makes binaries compatible between all the various
> stakeholders: compiler/tools people, distros, software vendors,
> etc. Lots of the rest of the details downstream of this can be
> changed, and people do this all the time - compare multilib to
> multi-arch for example. That all works fine *so long as* the runtime
> linker can be located and started OK.

The loader is still available via the old path, so external/third
party/local/other software works unchanged. This should negatively
only affect our 1st party packages, when running on a non-merged
distro.
And are _all_ our packages really 100% compatible with other distros
at all? Are they even supposed to be?

For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
machine, when I try to run it, it fails:

root@focal:/tmp# wget
http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
--2023-05-12 12:46:17--
http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27572 (27K) [application/vnd.debian.binary-package]
Saving to: 'efibootmgr_17-2_amd64.deb'

efibootmgr_17-2_amd64.deb
100%[===>]  26.93K
--.-KB/sin 0.04s

2023-05-12 12:46:17 (740 KB/s) - 'efibootmgr_17-2_amd64.deb' saved [27572/27572]

root@focal:/tmp# dpkg -x efibootmgr_17-2_amd64.deb ebm
root@focal:/tmp# ./ebm/bin/efibootmgr
./ebm/bin/efibootmgr: /lib/x86_64-linux-gnu/libc.so.6: version
`GLIBC_2.34' not found (required by ./ebm/bin/efibootmgr)

Should I file a severity: serious bug against efibootmgr because it is
not interoperable?

The answer is obviously not, because it would be absurd to expect a
binary compiled against libraries from one distro to "just work" on an
entirely different distro. Glibc itself is not forward compatible, and
is allowed to add new symbols, that are not present in older versions,
and packages are allowed to depend on them. Aren't those also ABI
breakages? What about all the libraries that bump soname? What about
binaries that rely on newer kernel interfaces, or IPC interfaces?

So, what I am asking is, what actual, real difference does it make if,
by default (and with an override available for example), packages
built on Debian for Debian record the ld path to point to its (actual)
location on Debian, via say a compiler spec file that is injected in a
deb build?
There very likely is some real difference and impact, and I am
genuinely and honestly asking what it could be. If nothing else, it's
an interesting topic, even if likely nothing comes out of it.

> Changing the interpreter path would mean moving to a Debian-specific
> ABI, breaking that compatibility. Hand-waving that away with (and I
> quote):
>
>   "The vast majority of distros today ship the loader in /usr/lib as
>   /lib is just a symlink, so it would be interoperable."
>
> is

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Fri, 12 May 2023 at 15:30, Steve McIntyre  wrote:
>
> On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> >On Fri, 12 May 2023 at 12:08, Steve McIntyre  wrote:
> >>
> >> On Fri, May 12, 2023 at 11:40:05AM +0100, Steve McIntyre wrote:
> >> >On Fri, May 12, 2023 at 10:49:32AM +0100, Luca Boccassi wrote:
> >> >>On Fri, 12 May 2023 at 09:40, Steve McIntyre  wrote:
> >> >>>
> >> >>> On Fri, May 12, 2023 at 07:40:00AM +0200, Ansgar wrote:
> >> >>> >
> >> >>> >The core issue as I see it is as follows:
> >> >>> >
> >> >>> >- Debian has decided to support only merged-/usr, including possibly
> >> >>> >  moving /bin/sh to /usr/bin/sh or using 
> >> >>> > /usr/lib*/ld-linux-x86-64.so.2
> >> >>> >  as the interpreter in binaries.
> >> >>>
> >> >>> WTF? *Nobody* has been talking about breaking ABI like this, that I've
> >> >>> seen. The interpreter must *not* be changed willy-nilly.
> >> >>
> >> >>Nothing's happening 'willy-nilly'. We are discussing a bunch of
> >> >>seemingly crazy options, as in, "what would _actually_ explode if we
> >> >>do this or do that?", on this very d-devel thread. I posted a longer
> >> >>version here some days ago:
> >> >>
> >> >>https://lists.debian.org/debian-gcc/2023/05/msg00030.html
> >> >
> >> >Oh holy fuck.
> >> >
> >> >You're talking about changing ABI by doing this. That *is* utterly
> >> >crazy. No.
> >>
> >> People have asked me to expand on this further...
> >>
> >> I've been involved in defining ABI before, specifically for
> >> armhf. It's not a quick and easy process. It needs buy-in from all
> >> sides to make things work, and people *really* value the
> >> interoperability that it enables.
> >>
> >> The interpreter path is one of the most important parts of the ABI
> >> spec, the bit that makes binaries compatible between all the various
> >> stakeholders: compiler/tools people, distros, software vendors,
> >> etc. Lots of the rest of the details downstream of this can be
> >> changed, and people do this all the time - compare multilib to
> >> multi-arch for example. That all works fine *so long as* the runtime
> >> linker can be located and started OK.
> >
> >The loader is still available via the old path, so external/third
> >party/local/other software works unchanged. This should negatively
> >only affect our 1st party packages, when running on a non-merged
> >distro.
>
> So why the hell do you want to break this in the first place? Does a
> symlink in the "wrong" place offend you for some reason? For that you
> want to change a core assumption in *every single binary* in Debian?
> Believe me, I've been here in the past when we made changes in armhf
> to accommodate earlier mistakes. That was just for one
> architecture. What possible benefit do you see in this change?

As it was mentioned on the list, because it makes bootstrapping
self-contained, that's a real and concrete benefit that some
developers like Helmut care greatly about, and that's why we are
talking about it. To me, it sounds very attractive to have a
self-contained and canonicalized distro-wide configuration. If the
canonical location where certain files are stored in /usr/bin or
/usr/lib, it seems sensible to me to configure Debian software to look
for it where we actually put it, while maintaining compatibility for
external/local software so that it keeps working. And it is also
unclear so far what would outright break - the externally defined ABI
in terms of where the loader can be accessed at, would still be
respected. Hence why questions are being asked. Nobody's being forced
to do anything, this is just a discussion.

> >And are _all_ our packages really 100% compatible with other distros
> >at all? Are they even supposed to be?
> >
> >For example, if I download efibootmgr from Bookworm on an Ubuntu Focal
> >machine, when I try to run it, it fails:
> >
> >root@focal:/tmp# wget
> >http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
> >--2023-05-12 12:46:17--
> >http://ftp.de.debian.org/debian/pool/main/e/efibootmgr/efibootmgr_17-2_amd64.deb
> >Resolving ftp.de.debian.org (ftp.de.debian.org)... 141.76.2.4
> >Connecting to ftp.de.debian.org (ftp.de.debian.org)|141.76.2.4|:80... 
&g

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Sat, 13 May 2023 at 14:59, RL  wrote:
>
> Luca Boccassi  writes:
>
> > I think documentation is fundamental for dealing with local
> > changes. When I say that I don't think we should worry about strange
> > local-only changes I mean exactly as you said, that I don't think we
> > should start shipping complicated code that tries to deal with people
> > diverting /sbin/init to /var/lolcat or whatever. Clear documentation
> > that tells "if you did X look at Y and do Z" is the bare minimum I'd
> > think.
>
> Hi,
>
> I have been trying to follow this discussion because I agree with the
> above, and we have been trying to get exactly this added to bookworm's
> release notes.
>
> Regardless of what happens in the future, users need to know what might
> break when the upgrade from a non-usr-merged system to bookworm. So far
> there isnt really anything clear out there for users that i can find
> that clearly lists the problems, but (as far as i can follow), the same
> issues in this thread may cause issues for people with more ususual
> setups.
>
> the merge request is here:
>   https://salsa.debian.org/ddp-team/release-notes/-/merge_requests/155/diffs
>
> (We are trying to give users a full list while also making it clear that
> most people wont have an issue on the upgrade)
>
> Would be great to confirm we've captured everything and given the right
> advice (to the extent that this is possible) on avoiding issues.

Thanks, looks good, left a couple of comments on the MR.

Kind regards,
Luca Boccassi

Re: dropping Priority field from binary packages for most packages

[Luca Boccassi]

On Sat, 13 May 2023 at 16:13, Ansgar  wrote:
>
> Hi,
>
> could we drop the Priority field from most packages? Most packages use
> "Priority: optional" and this is just noise in d/control (source
> package). Tools should just assume "optional" when no other value is
> set.
>
> Currently Policy documents Priority to be mandatory in 2.5:
>
> +---
> | Each package must have a priority value, which is set in the
> | metadata for the Debian archive and is also included in the
> | package’s control files (see Priority).
> +---[ https://www.debian.org/doc/debian-policy/ch-archive.html#priorities ]
>
> As well as only recommended:
>
> +---
> | The fields in this file are:
> | [...]
> | - Priority (recommended)
> +---[ 
> https://www.debian.org/doc/debian-policy/ch-controlfields.html#binary-package-control-files-debian-control
>  ]
>
> I would like to drop it pretty much everywhere, most importantly
> debian/control in source packages (as often humans edit these). But it
> could be dropped in other places (CONTROL in .deb and Packages indices)
> as well.

+1 for decluttering, especially human-maintained metadata files.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
>
> On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> > The loader is still available via the old path, so external/third
> > party/local/other software works unchanged. This should negatively
> > only affect our 1st party packages, when running on a non-merged
> > distro.
> > And are _all_ our packages really 100% compatible with other distros
> > at all? Are they even supposed to be?
>
> People build things on Debian that are not Debian packages. People
> compile binaries on Debian, and expect them to work on any system that
> has sufficiently new libraries.
>
> This is *not* about Debian packages failing to work on other
> distributions; this is about *software compiled on Debian* faliing to
> work in other environments.

Why would "software compiled on Debian" fail to work in other
environments? Well, there are many reasons actually, people invented
containers/flatpaks/snaps exactly for that reason. But nothing do with
anything discussed here though, as far as I can tell?

> If you build a dynamically linked binary that only depends on glibc, you
> can expect it to be reasonably portable, to any system that uses glibc
> and has a sufficiently new version.
>
> Debian stable is, in fact, one of the common environments people use to
> compile binaries for distribution.

"sufficiently new version" is doing a lot of work there. We have
shlibs dependencies for a reason. In fact, the most common environment
used to distribute binaries is the EOL Ubuntu 16.04, slowly switching
to the soon-to-be-EOL 18.04. glibc is not forward-compatible, new
symbols are added all the time and are used all the time, and you
don't jump on a brand new distribution to do that kind of work, that
would be self-defeating.

> > So, what I am asking is, what actual, real difference does it make if,
> > by default (and with an override available for example), packages
> > built on Debian for Debian record the ld path to point to its (actual)
> > location on Debian, via say a compiler spec file that is injected in a
> > deb build?
>
> Making binaries built *on* Debian different than binaries built *for*
> Debian would introduce a needless additional source of complexity,
> compared to just compiling code the same way in both cases.

That's not how it works today already. There are several significant
differences between just running "gcc sources.c" and building a
package via debhelper on a buildd, they are not the same thing at all,
and they haven't been since forever, there are dozens of
compiler/linker options that the Debian package build environment
sets. Or will you now also ask the distribution to rollback multiarch,
hardening, SOURCE_DATE_EPOCH, -ffile-prefix-map and all the other
reproducibility options, and so on? These and many more are all
"needless additional sources of complexity, compared to just compiling
code the same way" too. Because guess what, there are people who
couldn't possibly care less about
multiarch/security/reproducibility/etc, and there will also be a
subset of users who considers a subset of those compiler options
"needless". So are you going to push to have all of that reverted? And
also are you going to propose a Policy change that forbids adding any
new compiler/linker option to the package build process?

> To frame this in different terms: consider that one of the major goals
> of systemd has been to harmonize across distributions and eliminate
> needless variations that don't serve much actual purpose (e.g.
> variations in config file paths for the same config file). Consider how
> much effort systemd went to work with distributions, understand and deal
> with the *important* variations, and try to convince them to abandon the
> *unimportant* variations. Now imagine if someone came along and said
> "let's patch systemd to put unit files in /purple/; it'll work with
> everything in our distribution".

Pretty sure the Nix folks are already doing pretty much that. And if
it works for their case, all the power to them.

> Or, imagine if someone said "let's inject an argument to gzip, only for
> building the .gz files sihpped in our packages of course, to modify the
> gzip header and remove a few of the extraneous additional fields; it'll
> be fine, because we've patched our gzip to parse it"

Not really related, archives are _intended_ to be opened anywhere for
any reason. Do you have any actual related use case that would no
longer work? Because that would be the easiest and most convincing
counter-factual that could be provided.

> The x86-64 ABI is set. Feel free to make the case to the next
> architecture designer that their new ABI should ha

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 01:07, Peter Pentchev  wrote:
>
> On Mon, May 15, 2023 at 12:24:15AM +0100, Luca Boccassi wrote:
> > On Sun, 14 May 2023 at 22:37, Josh Triplett  wrote:
> > >
> > > On Fri, May 12, 2023 at 01:11:38PM +0100, Luca Boccassi wrote:
> > > > The loader is still available via the old path, so external/third
> > > > party/local/other software works unchanged. This should negatively
> > > > only affect our 1st party packages, when running on a non-merged
> > > > distro.
> > > > And are _all_ our packages really 100% compatible with other distros
> > > > at all? Are they even supposed to be?
> > >
> > > People build things on Debian that are not Debian packages. People
> > > compile binaries on Debian, and expect them to work on any system that
> > > has sufficiently new libraries.
> > >
> > > This is *not* about Debian packages failing to work on other
> > > distributions; this is about *software compiled on Debian* faliing to
> > > work in other environments.
> >
> > Why would "software compiled on Debian" fail to work in other
> > environments? Well, there are many reasons actually, people invented
> > containers/flatpaks/snaps exactly for that reason. But nothing do with
> > anything discussed here though, as far as I can tell?
>
> If an ELF executable, compiled on Debian, records its interpreter as
> /usr/lib/ld-linux.so.2, what happens when one tries to run it on
> a non-usr-merged system? Even one with a recent enough glibc version?

This is not about locally built ELF executables, no difference in those.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 01:14, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > Why would "software compiled on Debian" fail to work in other
> > environments? Well, there are many reasons actually, people invented
> > containers/flatpaks/snaps exactly for that reason. But nothing do with
> > anything discussed here though, as far as I can tell?
>
> My understanding is that this specific thread is about a mention that we
> may want to change PT_INTERP to /usr/lib64/ld-linux-x86-64.so.2 or some
> similar path.
>
> If PT_INTERP points to a file that doesn't exist, the program is obviously
> not going to run.  The Linux x86_64 ABI says it must point to
> /lib64/ld-linux-x86-64.so.2.  If we build binaries that use some other
> value, then we are not building ABI-compliant binaries and they may not
> run on other systems.  This is the whole point of an ABI.

This is not about locally compiled software or such, only packages
(and maybe even just a subset of them).

> An obvious specific example of such a system would be one that didn't
> merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other
> path, but that's just one obvious example.  There may be others; the whole
> point of an ABI is that you do not change things like this, not even if
> you can't personally imagine why your change wouldn't be harmful.  There's
> a whole process for changing an ABI that involves everyone else agreeing
> as well, and unless one goes through that process, the ABI is what it is.
> Debian not building ABI-compliant binaries would be highly surprising.

That's self-evidently not true, as there are other distributions where
that already happens, it's been already mentioned. Besides, we are not
talking about sacred religious texts - the point is making things
work. If they do, is it _really_ non-compliant/incompatible?

> Incidentally, that remains true even if we only do that in distribution
> packages.  I certainly have copied binaries from a Debian package to other
> Linux systems before for various reasons and expected them to run.  Sure,
> this might not work for other reasons outside of our control, but that's
> no reason to be gratuitously incompatible by breaking the ABI,
> particularly for what seem to be annoyances of our own creation with known
> workarounds.

Thanks, that's the first actual real example mentioned so far. And
it's an interesting one: taking a $random Debian package and using it
on a completely different, non-Debian system. Is that a supported use
case? If so, does that mean that I can go ahead and raise a Severity:
serious bug on any package that doesn't work in such a way?

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 02:26, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > That's self-evidently not true, as there are other distributions where
> > that already happens, it's been already mentioned.
>
> You've mentioned this a couple of times but I don't think I've seen the
> message where the details were explained.  Maybe this was only in your
> message posted to debian-gcc, which wasn't part of this thread?  (It's
> also possible that I just missed it somewhere.)
>
> That message only mentions GUIX, which I don't know very much about, but
> my recollection (maybe wrong?) is that it's a NIX variant that is doing
> special tricks to support immutable package trees and
> roll-forward/roll-back upgrades.  I can see why that might be motivation
> to build incompatible binaries in order to preserve some other invariant
> they're trying for as the point of their distribution (in particular, I
> suspect they're pinning binaries to a specific version of the dynamic
> loader as part of the whole immutable tree strategy).  That's a perfectly
> fine decision in a distribution that's trying to do something very
> different and is a bit of a science experiment, but I don't think that
> describes Debian.
>
> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh, major
> player in Linux distributions, and I'm not sure how much they care about
> compatibility with anyone else.)

This is a counter-example to confute the assertion that *everybody*
does the same thing, which has been made multiple times. I'm not sure
whether it's an experiment or not, I mean if you ask their
users/developers I think they'd tell you it's very much production
ready, but it is largely irrelevant: it exists, and that's the only
reason why it was mentioned, as it shows that it is _possible_ to do
that and be a working distribution. Doesn't imply it's automatically
desirable, but that was not the intention.

> > Besides, we are not talking about sacred religious texts - the point is
> > making things work. If they do, is it _really_
> > non-compliant/incompatible?
>
> I understand your point in making this argument, but please understand
> that this sort of willingness to change things if one didn't think they
> would cause problems didn't work very well, and was part of what led to
> the development of standardized ABIs in the first place.  Those of us who
> have been around longer than Linux have ABIs have a bit of a strong
> reaction here (I think this is also what you're seeing from Steve),
> because we remember the bad old days.  I still have compatibility code
> around to handle the fact that gcc on IRIX miscompiled calls to inet_ntoa
> because gcc didn't correctly implement the IRIX ABI.
>
> People are very bad at judging whether their new idea would be *really*
> incompatible.  This is why these days everyone tries to follow the ABI
> pretty closely.
>
> And in any case, changing PT_INTERP is trivially and obviously
> incompatible; the binary will simply not run on a system that doesn't have
> that path.  So it's not like we have to carefully judge nuance here.  Your
> argument, so far as I can tell, is basically "but no one will ever want to
> run those binaries on a non-/usr-merged system anyway," which is basically
> conceding the incompatibility point since the ABI doesn't require merged
> /usr.

Not quite: my argument is that binaries from these packages are not
intended and not required to be ran on non-Debian systems, so there's
no incompatibility introduced in the first place - everything still
works where it is supposed to, exactly as it was before.

> There's also some other history here: Debian is not super-happy with the
> PT_INTERP because ideally we'd prefer it use a path compatible with our
> multiarch approach.  I believe we raised that and no one had any interest
> in trying to change anything, so we lived with the limitations that
> creates.  (And I think that was the right decision.)
>
> > Thanks, that's the first actual real example mentioned so far. And it's
> > an interesting one: taking a $random Debian package and using it on a
> > completely different, non-Debian system. Is that a supported use case?
> > If so, does that mean that I can go ahead and raise a Severity: serious
> > bug on any package that doesn't work in such a way?
>
> I feel like you're distorting my argument here to try to make some sort of
> slippery slope argument, and it's coming across as possibly more
> aggressive than you had intended.

No aggression intended whatsoever, sorry if it appeared that way.

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 14:36, Steve McIntyre  wrote:
>
> Hey Johannes,
>
> On Mon, May 15, 2023 at 06:48:04AM +0200, Johannes Schauer Marin Rodrigues 
> wrote:
> >So did we not years ago decide, that the result of the "cross- and
> >inter-project discussion" is, that everybody is going merged-/usr and that's
> >why we need it too and that's why it is okay to build a system where binaries
> >and scripts built for it just may not run on those other systems that do not 
> >do
> >it?  With merged-/usr we already *did* "change fundamental things around" for
> >reasons that are really not clear to me (but which i do not want to discuss
> >here) and as a result did not "care about interoperability" (with those who 
> >do
> >not also adopt it). In my own Debian work I so far only got extra work 
> >because
> >of merged-/usr and I do not see the benefits (yet) and I was hoping that
> >"changing fundamental things around Linux and (basically) not caring about
> >interoperability" was *not* Debian's attitude but alas here we are.
> >
> >So have we not already burned the bridges to the non-merged-/usr world? Why 
> >was
> >it okay back then to say "we can make this change because all other important
> >players are doing merged-/usr so we can/have to as well". And now in the
> >PT_INTERP discussion somehow we care again about those systems? I thought we
> >already had the "cross- and inter-project discussion" about merged-/usr and
> >because the result was "yes, go for it" we did it too. But if that is the 
> >case,
> >why do we now care for a subset of the interoperability problems caused by
> >merged-/usr for systems that don't have it?
>
> This change is absolutely *not* needed to make merged-/usr work; if
> anybody is claiming that it is, then they are not being 100% honest
> with us. All the other distros doing merged-/usr have done it without
> making this change, and it's also been working OK for us so far
> without this change.

That is absolutely true, it is not mandatory. It is one possible
solution (of many) to a particular use case being sounded out, that's
all. I don't think it was mentioned by anybody as needed, if it was,
happy to clarify.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 16:18, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
> > On Mon, 15 May 2023 at 02:26, Russ Allbery  wrote:
>
> >> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh,
> >> major player in Linux distributions, and I'm not sure how much they
> >> care about compatibility with anyone else.)
>
> > This is a counter-example to confute the assertion that *everybody* does
> > the same thing, which has been made multiple times. I'm not sure whether
> > it's an experiment or not, I mean if you ask their users/developers I
> > think they'd tell you it's very much production ready, but it is largely
> > irrelevant: it exists, and that's the only reason why it was mentioned,
> > as it shows that it is _possible_ to do that and be a working
> > distribution. Doesn't imply it's automatically desirable, but that was
> > not the intention.
>
> Ah, okay, I'm happy to agree with that point: you can violate the ABI and
> continue to be a working distribution.  (There are a lot of parts of the
> ABI that if you violated them you would not have a working distribution,
> but this is not one of them so far as I can tell.)
>
> > Not quite: my argument is that binaries from these packages are not
> > intended and not required to be ran on non-Debian systems, so there's no
> > incompatibility introduced in the first place - everything still works
> > where it is supposed to, exactly as it was before.
>
> I think we're saying the same thing but quibbling over phrasing.  I'd put
> that as saying that it's fine for the binaries of certain core Debian
> packages to be incompatible with the ABI because they're not intended to
> be used outside of Debian.  (In other words, I'm talking about
> incompatibility as a concrete, testable property of a binary, and I think
> you're talking about incompatibility as a more abstract concept of a
> distribution.)
>
> > No aggression intended whatsoever, sorry if it appeared that way. I am
> > trying to understand what the rules are.
>
> Well, the rule that I'd ideally set is don't break the ABI, even if it's
> not obvious why breaking the ABI is a bad idea or you can't see any bad
> consequences that could come from it, unless the reason for breaking the
> ABI is absolutely central to the mission and purpose of Debian.
>
> That said, it's not like we've never shipped a binary in Debian with a
> different PT_INTERP.  (I vaguely remember that some programming language
> uses PT_INTERP tricks for some sort of private binary scheme?  Objective
> CAML or something?  I ran across it once years ago and can't remember the
> details.  Also, IIRC klibc does some sort of PT_INTERP trick in some
> situations that I don't remember the details of, although I don't think it
> does that with general binaries.)  So I do see your point that you would
> prefer the rule to be more pragmatic than that.
>
> My counterargument is that this proposal seems to mostly be about avoiding
> having to create a symlink at a critical point in the bootstrap process,
> and while it's tricky to get the timing right (and therefore kind of
> annoying), the resulting usable system has to have that symlink anyway (I
> think there's no disagreement about that).  Not following the ABI for core
> binaries seems like a scary change with unknown consquences to a bunch of
> core packages to solve what looks like a relatively minor (if admittedly
> annoying!) problem.
>
> Note that the target of PT_INTERP on Debian is *already* a symlink, to
> /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 which was the multiarch path
> that we actually want to use.  We're already ensuring compatibility with a
> symlink and I think we should just keep doing that and not venture into
> these waters.

It's not even a proposal, it's a discussion to see "what would
actually break if X happened". That's why I'm asking for concrete use
cases, rather than theoretical points. Also, I am interested in what
the rules are. For example, glibc compatibility is just as important
if not more, why does that follow a different standard? If anything,
adding a missing symlink is trivial and a single command. Adding a
missing symbol to a shared object... not quite. There is an endless
list of downstream-only debianisms that break cross-compatibility in
just the same way (if not worse), and yet nobody cares about those.
Why?

> >> The world does not divide neatly into supported and unsupported use
> >> cases.  There are a lot of things I do to computers that I expect to
> >> work in some situa

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Mon, 15 May 2023 at 18:54, Simon McVittie  wrote:
>
> On Sun, 14 May 2023 at 23:37:34 +0200, Josh Triplett wrote:
> > People build things on Debian that are not Debian packages. People
> > compile binaries on Debian, and expect them to work on any system that
> > has sufficiently new libraries.
>
> *raises hand*
>
> Hello, I represent an example of those people. With my $work hat on,
> I'm involved in maintaining a family of Debian derivatives (the Steam
> Runtime) that is used to run Steam games on arbitrary distributions,
> including but not limited to Debian. Some of these binaries are built
> on a Debian derivative, but run on an arbitrary other distribution,
> using a RPATH[1] to find their non-glibc dependencies.
>
> At the moment those binaries are built in ancient environments (based
> on Ubuntu 12.04 and Debian 8), but as newer versions of glibc become
> ubiquitous, we'll want to start relying on newer versions of Debian
> (which will still be very old versions *at the time*, but I'm sure that
> by 2030 or 2040 we'll want to be using versions of Debian that, in 2023,
> are not released yet). In this use-case, yes we do need to be using the
> interoperable ELF interpreter path.
>
> We were able to use (Ubuntu and) Debian for this *because* Debian is
> a relatively "ordinary" distribution that tends to follow cross-distro
> standards. The major counterexample is multiarch library paths, but
> multiarch library paths have some compelling technical advantages, and
> because there's no ambiguity about which architecture uses which directory,
> they're actually better for interop in some ways than /usr/lib (which
> is ambiguous, because the Red Hat family of distros expects to find 32-bit
> libraries there, but the Arch family expects 64-bit libraries, and it's
> not possible to construct a tree where both get what they want).
>
> I've spent a lot of the last 5 years working on putting Steam games in
> containers so that they'll work more reliably on all distros, including
> Debian, with less reliance on weird library search paths (although we
> still have to use binaries built on an ancient Debian derivative with a
> non-trivial RPATH to bootstrap the container environment). Because of
> constraints around recent GPUs needing current graphics drivers even
> if running on an otherwise ancient library stack, Steam's container
> runtime has to construct a hybrid environment where glibc is either the
> one from the host or the one from the container runtime environment,
> whichever is newer; and while doing that, we have experienced some
> broken situations that were caused by distributions diverging from the
> interoperable ELF interpreter. One concrete example is that Arch Linux
> uses the interoperable ELF interpreter for *almost* all executables,
> but uses a different ELF interpreter for executables from the glibc
> source package, for whatever reason; we were able to work around this,
> but for a while it caused Arch to fail to launch these containers where
> Debian/Fedora/etc. could.

This sounds like a very interesting use case, and the first real one
mentioned, which is great to see - but I do not fully follow yet, from
what you are saying it seems to me that what you need is for your
binaries to use the usual pt_interp, that bit is clear. But why does
it matter if /usr/bin/ls on the host uses a different one? It's your
executables that you ship as part of that runtime that are the entry
points that need the usual loader path for your chroot-on-steroids,
no? The loader would still be reachable as it always was in this
theoretical exercise. I am probably missing something in how this
works in details.

Kind regards,
Luca Boccassi

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Tue, 16 May 2023 at 04:22, Russ Allbery  wrote:
> Luca Boccassi  writes:
> > On Mon, 15 May 2023 at 16:18, Russ Allbery  wrote:
>
> >> Note that we're not talking about complicated packages with lots of
> >> runtime like, say, Emacs.  As I understand it your proposal wouldn't
> >> change PT_INTERP for that binary anyway.  We're presumably talking
> >> about the kind of binaries that you need to bootstrap a minimal system,
> >> so packages like coreutils or bash.  And I would indeed expect those
> >> binaries to be generally portable, as long as the same critical shared
> >> libraries are available on other systems (in this case, PCRE2 and
> >> ncurses).
>
> > Is that really the case? Let's test that hypothesis:
>
> I think you may have not read my whole message before replying, and also
> please assume that I know really basic facts about glibc compatibility and
> am not referring to that.
>
> I said "of a similar vintage" (farther down in my message) because of
> course we all know that binaries built against newer versions of glibc
> don't run on systems with older versions of glibc (and likewise for shared
> libraries in general and thus libselinux), and you tested a Debian
> unstable package on an Ubuntu system from 2020.  This says nothing
> interesting and has nothing to do with my point.

It does say something interesting. When we started, the assertion was
that packages not relying on the symlink being present was fundamental
for portability and cross-compatibility. Then, it shrinked to
portability and cross-compatibility of a subset of packages. Now it
further shrank to portability and cross-compatibility of a subset of
packages of a subset of vintages.
Why is the requirement that libselinux and glibc are not too recent
fine to have on coreutils, but the requirement that there's a symlink
is out of the question? If anything, the latter is trivial to add, so
the formers should feel more "disruptive" and game-breaking, no?

> > Whoops. Does this make coreutils rc-buggy now? ;-)
>
> You are the only person who is talking about RC bugs.  The bar here is not
> "prove to me that this is RC buggy," the bar is "you have to prove to a
> bunch of Debian core maintainers that they should break the ABI in their
> packages" (not to mention the additional small but permanent build
> complexity).  Demanding they prove to you that it's a bad idea is not how
> this works.

It's a tongue-in-cheek comment, I had hoped the emojii would give that away.

> The point of standards like an ABI is that a bunch of entirely unrelated
> people who never talk to each other and never look at each other's
> software are allowed to rely on them and assume no one else will break
> them.  This is how free software scales; without invariants that everyone
> can rely on without having to explain how they're relying on them, it is
> much more difficult to get an ecosystem to work together.  We don't just
> break things because they don't seem important; the space of people who
> may be relying on this standard is unknowable, which is the entire point.
> Opening those boxes is really expensive (in time, planning, communication,
> debugging, and yes, compatibility) and we should only do it when it
> really, really matters.

But does it really work, or do we just hope it does? I personally see
a great deal of signals that say it doesn't. For one, in 2023, "build
one binary and run it everywhere" doesn't really seem to me to be the
principal choice in terms of portability. It might have been once, and
there might be a number of relevant use cases still, but it seems to
me the industry has largely gone in a very different direction
nowadays.

> > It did look like a veto to me. More importantly, isn't relying on
> > passersby to spot alleged harmful changes dangerous, especially for
> > undocumented, uncodified and untested use cases, like unspecified and
> > vague cross-compatibility requirements?
>
> I'm honestly boggled.  This is a thread on debian-devel, which is
> literally how we do architecture vetting in this project.
>
> I absolutely do not think that we can write down everything of importance
> in designing a distribution so that we don't have to have conversations
> with other people in the project who have deep expertise when considering
> a significant architectural change like changing the PT_INTERP path of
> core binaries.

Not everything of course, but wouldn't it be worth it to specify core
items such as cross-compatibility requirements? Not in terms of
implementation details, but in terms of what the goals and the minimum
expectations are. The

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Tue, 16 May 2023 at 09:27, Simon McVittie  wrote:
>
> On Tue, 16 May 2023 at 02:50:48 +0100, Luca Boccassi wrote:
> > This sounds like a very interesting use case, and the first real one
> > mentioned, which is great to see - but I do not fully follow yet, from
> > what you are saying it seems to me that what you need is for your
> > binaries to use the usual pt_interp, that bit is clear. But why does
> > it matter if /usr/bin/ls on the host uses a different one?
>
> We don't need to run the ls from the host, but we do need to run
> glibc-related executables like ldconfig and localedef from either the
> host or the container runtime, whichever is newer. Because glibc is
> a single source package, executables and libraries within the glibc
> bubble sometimes make use of private symbols in libraries that are also
> within the glibc bubble (and IMO they have a right to do so), even though
> executables from outside glibc would be discouraged or disallowed from
> doing so. This means that when we have chosen a particular version of
> glibc (which, again, must be whichever one is newer), we try to use its
> matching version for *everything* originating in the glibc source package.
>
> In principle we could get exactly the same situation if we've imported a
> library from the host system (as a dependency of the graphics stack) that
> calls an executable as a subprocess and expects it to be >= the version
> it was compiled for - hopefully not (/usr)/bin/ls, but potentially others.

Thanks for the clarification, so if I understood correctly, your use
case is that sometimes (eg: when they are newer) you pull binaries
(eg: ldconfig) from the host, and run them from the container? So, in
case let's say ldconfig on the host points to /usr/lib/ld, but because
your container is not usr-merged, it wouldn't find the interpreter and
fail?

> The wider point than my specific use-case, though, is that when there's a
> standard, you can't predict what other software authors have looked at the
> statement "you can rely on this" and ... relied on it. See also Russ's
> (as ever, excellent) mails to the same thread.
>
> I appreciate that you are trying to explore the edges of the
> problem/constraint space and say "what if we did this, could that work?",
> and it's good that you are doing that; but part of that process is
> working with the other people on this list when they say "no, we can't
> do that because...", and respecting their input.

I respect and appreciate the input, but I want to understand it too,
hence the "because..." part is what I was looking for - so thanks for
providing it, it is really useful.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Wed, 17 May 2023 at 10:31, Helmut Grohne  wrote:
>
> Hi,
>
> This bootstrap aspect got me and I discussed this with a number of
> people and did some research.
>
> On Sun, May 07, 2023 at 12:51:21PM +0100, Luca Boccassi wrote:
> > I don't think this is true? At least not in the broader sense: if you
> > compile something on Debian, it will obviously get linked against
> > libraries and dependencies as they are in Debian.
> > Perhaps what you mean is that, given an entire separate sysroot-like
> > tree, passing the appropriate compiler and linker flags and
> > environment variables, you can use the local compiler we ship to build
> > 'foreign' programs. That is true, but again it requires to set up the
> > environment appropriately, including linker flags. And the caller
> > needs to ensure the environment, including linker flags, is
> > appropriate for the target environment (I guess 'host' environment, in
> > GNU parlance). Therefore, I don't think it would be unreasonable to
> > require that if the target environment is split-usr, then the caller
> > also needs to specify an appropriate
> > '-Wl,--dynamic-linker=/lib/ld-whatever' option.
>
> Given the feedback, I am convinced that changing PT_INTERP is a stupid
> idea regardless of whether it is technically feasible. There must be a
> better way. Let's step back a bit.
>
> The underlying problem here is performing the initial filesystem
> bootstrap. The semantics of this are a bit vague as they are not spelled
> out in policy, so we will have to derive them from implementations.
>
> I think the major players are (in descending popularity):
>  * debootstrap
>  * mmdebstrap
>  * cdebootstrap
>  * multistrap
>
> multistrap predates mmdebstrap and when there was no mmdebstrap, I used
> it a lot. When attempting to test it, I totally couldn't convince it to
> bootstrap from an unsigned or locally signed repository.  The patch in
> #908451 didn't cut it. I also note that it creates a /lib64 -> /lib
> symbolic link which feels quite incompatible with merged-/usr.  For
> these reasons, I am dropping multistrap from the tools under
> consideration and recommend removing it from the archive. If you happen
> to use multistrap, now would be a good moment to tell me. Personally,
> all of my use cases of multistrap have been converted to mmdebstrap and
> that made a lot of things simpler.
>
> cdebootstrap vaguely works though unsigned operation seems dysfunctional
> as it runs apt-get update during cdebootstrap-helper-apt.postinst and
> that fails. I happen to not have figured out why and treat this failure
> as a success.
>
> So the most popular implementations quite evidently are debootstrap and
> mmdebstrap and both "just work". I note though that they work quite
> differently:
>  * debootstrap (depending on flags including --variant) pre-merges its
>chroot while mmdebstrap relies on packages doing it.
>
>I think that the question whether a distribution is merged is a
>property of the distribution and not the bootstrap tool, so I
>strongly recommend following mmdebstrap's view on this. The
>debootstrap way means that we have to include patches for every
>derivative, which is a process that does not scale well.
>
>  * mmdebstrap operates in two phases. It first unpacks and configures a
>rather minimal set of packages and then proceeds to adding packages
>passed to --include in a second phase once essential is fully
>configured while debootstrap immediately unpacks everything.
>
>I think the debootstrap approach is slightly worse here, because it
>means that preinst scripts of non-essential packages cannot rely on
>essential packages having been configured.
>
> In any case, we have to deal with both behaviours.
>
> After this little excursion into bootstrap technology, let's go back to
> the /usr-merge and its effects.
>
> I think at this point, we have quite universal consensus about the goal
> of moving files to their canonical location (i.e. from / to /usr) as a
> solution to the aliasing problems while we do not have consensus on
> precisely how to do this (i.e. with changing dpkg or without). If you
> believe that this is not consensus, please speak up.
>
> So in a distant future our packages will not contain any files in /bin
> or /lib. In particular, this affects /bin/sh and the dynamic loader,
> both of which are required to run maintainer scripts, which are
> currently required for creating the symbolic links. Boom.
>
> Solutions have been proposed to this and I think they all fall into one
> of the following four cate

Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)

[Luca Boccassi]

On Wed, 17 May 2023 at 01:05, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
>
> > It does say something interesting. When we started, the assertion was
> > that packages not relying on the symlink being present was fundamental
> > for portability and cross-compatibility. Then, it shrinked to
> > portability and cross-compatibility of a subset of packages. Now it
> > further shrank to portability and cross-compatibility of a subset of
> > packages of a subset of vintages.
>
> I think it's pretty clear that I'm totally failing to communicate what I
> was getting at with this example and the process is becoming incredibly
> frustrating, so I'm going to stop trying here.
>
> > But does it really work, or do we just hope it does? I personally see a
> > great deal of signals that say it doesn't. For one, in 2023, "build one
> > binary and run it everywhere" doesn't really seem to me to be the
> > principal choice in terms of portability.
>
> Well, believe what you believe, but I literally do that daily, as does
> anyone else who regularly uses software from a Rust or Go ecosystem.  Not
> a single work day goes by without me running, on some random Ubuntu or Red
> Hat or Debian system, binaries that were compiled on some random other
> Linux distribution (often I have no idea which one).
>
> > It might have been once, and there might be a number of relevant use
> > cases still, but it seems to me the industry has largely gone in a very
> > different direction nowadays.
>
> I do think the industry is moving away (well, has already moved away) from
> Linux Standards Base pre-compiled C binaries without wrappers like snap or
> flatpak, although there are some very notable exceptions, such as Steam's
> special use cases explained elsewhere on the thread.  I don't believe the
> industry is moving away from Go and Rust binaries, so this is, I think, a
> more complicated picture than your description indicates.

I know absolutely nothing about Go, and preciously little about Rust,
but my understanding from news article and so on was that both of
those ecosystems pretty openly avoid "traditional" distributions, in
favour of their own self-contained ecosystems. So would that use case
be affected, even by an hypothetical and at this point extremely
unlikely archive-wide change? Meaning, are those randomly compiled
Go/Rust binaries you run for work taken from the Debian archive, or
locally/externally/self compiled? I do know we do ship a number of
those, although I am very oblivious to the number and nature of that
subset. And conversely, if the hypothetical change was just to the
essential set, I assume there's no Go/Rust there, right?

> > Not everything of course, but wouldn't it be worth it to specify core
> > items such as cross-compatibility requirements? Not in terms of
> > implementation details, but in terms of what the goals and the minimum
> > expectations are. There are obviously some, and they are obviously of
> > great importance to many, and yet to me it feels like they shift every
> > time I ask a different question.
>
> I am certainly never going to say no to more maintained documentation.
> That would be great.  If you're feeling motivated and you're willing to
> really listen to what people are saying and accept and understand a lot of
> ambiguity and a lot of assumptions that don't match your assumptions, go
> for it!
>
> I personally am not even doing justice to Policy in its existing scope, so
> this isn't something I will personally be tackling.  Honestly, I should
> have spent all of the time I spent on this thread working on Policy
> instead.  :)

Sure, I can add that to my todo list. But I have already two policy
items open for review, so certainly won't get to it before those are
sorted, hint hint :-)
(probably it would have been best not to mention it, lol)

> > But if you prefer to focus on first principles, that's great! I've been
> > asking this a lot: is cross-distribution harmonization a core
> > requirement? Is it important enough to trump other goals and
> > distro-local benefits? If so, isn't it worth to discuss it and then
> > pencil it down?
>
> I think some parts of it are a core requirement.  It would be very
> surprising, and very bad, if we couldn't run Go and Rust binaries built on
> another distribution, for example, or if Go or Rust binaries built on
> Debian couldn't be used on other distributions, both subject to the normal
> constraints of glibc cross-compatibility that everyone building binaries
> already knows about.  I think other parts of it are not a core
> requirement, but still something t

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Thu, 18 May 2023 at 07:39, Simon Richter  wrote:
>
> Hi,
>
> On 5/18/23 02:15, Sam Hartman wrote:
>
> >  Helmut> I think at this point, we have quite universal consensus
> >  Helmut> about the goal of moving files to their canonical location
> >  Helmut> (i.e. from / to /usr) as a solution to the aliasing problems
> >  Helmut> while we do not have consensus on precisely how to do this
> >  Helmut> (i.e. with changing dpkg or without). If you believe that
> >  Helmut> this is not consensus, please speak up.
>
> > I agree we have strong consensus that we want to move files to their
> > canonical locations.
>
> > I'm not entirely sure I'd agree that we have consensus that's our
> > solution to the aliasing problem.
>
> It's the other way around: moving the files as a solution to the
> aliasing problem is the strongest argument in favour of moving the files
> inside the packages.
>
> Without it, leaving them in place makes no difference for usrmerged
> systems, and allows derived distributions that don't need usrmerge to
> continue using our packages.

Not quite. Having packages only ship files under /usr (and possibly
/etc) is very much a goal in itself for a lot of us.

> > If for example we accomplish the move to canonical locations by changing
> > dpkg, we might well get some form of aliasing support in dpkg.
>
> IMO, that is still the preferred solution:
>
>   - it is actually safe, because dpkg knows what is going on and can
> reject conflicting changes
>   - there is no guarantee that usrmerge will be permanent or the last
> transition of this kind

It is permanent, there are several upstream projects that will drop
support for legacy layouts very soon, and it will not be re-added
back. This will become more and more common, as simply most will stop
caring and paying any attention to this detail. Debian is pretty much
the last relevant holdout here, and that's going to end in a couple of
weeks.

>   - it also solves the bootstrap problem

It also is the least likely to succeed, and the most likely to cause
significant "social" upheavals.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Fri, 19 May 2023 at 01:30, Simon Richter  wrote:
>
> Hi,
>
> On 5/18/23 18:08, Luca Boccassi wrote:
>
> >> Without it, leaving them in place makes no difference for usrmerged
> >> systems, and allows derived distributions that don't need usrmerge to
> >> continue using our packages.
>
> > Not quite. Having packages only ship files under /usr (and possibly
> > /etc) is very much a goal in itself for a lot of us.
>
> My point is: that is not an end goal, because it provides no real
> tangible benefit on its own.

For yourself. Again, for others, it is and it does.

> It does make sense in the context of building immutable images, which is
> *also* a bootstrapping problem, and probably worth being supported with
> proper tooling.
>
> >>- there is no guarantee that usrmerge will be permanent or the last
> >> transition of this kind
>
> > It is permanent, there are several upstream projects that will drop
> > support for legacy layouts very soon, and it will not be re-added
> > back.
>
> You are currently building a "legacy" system, it will just take a bit of
> time to reach that status. The less you anticipate future needs, the
> faster this will be.
>
> I understand that you are also a member of one of these upstream
> projects, and that you are taking the interests of this project to
> "pretty much the last relevant holdout" here. Has it occurred to you
> that you are also wearing a Debian hat, and you could be taking the
> interests of the Debian project to said upstream project?

Has it occurred to you that there might be a specific reason why said
upstream project has kept compatibility with legacy cruft that only
benefits Debian for so many years, at non-zero cost for developers,
despite everything else that's relevant having long since moved on? It
is really not difficult to find out what that reason might be, and
would have been better to do so before throwing around such
accusations. And if you can't find it, you can always go to one of the
numerous available channels and ask other maintainers. Why don't you
do that, ask how come support for Debian stable for this and many
other aspects is kept intact and who's behind that effort, and report
back the answer?

> >>- it also solves the bootstrap problem
>
> > It also is the least likely to succeed, and the most likely to cause
> > significant "social" upheavals.
>
> The only social problem I see is that you are trying to create a
> situation in which other people are compelled to do your work for you if
> they want it to be done properly.

No, the social problem is that there is one maintainer who is allowed
to ignore the TC and roadblock the entire distribution, so that
something that's been trivially and quickly done pretty much literally
everywhere else is instead made incredibly difficult and long-winded.
But despite that, we are getting there, slow and steady, and
remarkably well given the difficult circumstances outwith our control.

> So far, you have been throwing out "solutions", and left the analysis of
> the feasibility of those to other people, then, after three iterations,
> you demanded a full write-up of all existing use cases and blanket
> permission to ignore anything not brought up in this list.

I have literally no idea what you are talking about. I have
contributed what I can in my spare time, unblocking the transition
last year and helping out Helmut and others this year. This is not my
day job, by the way. What have you done to help, precisely?

> The thing is: I see more enthusiasm and self-directed problem solving
> skills from the interns at the company where I work, and at the same
> time you are one of the top contributors of the upstream project whose
> ideas of a "supported" configuration we are supposed to follow.

I seriously do not appreciate your tone, you are out of line. Please stop.

Kind regards,
Luca Boccassi

Re: i386 in the future (was Re: 64-bit time_t transition for 32-bit archs: a proposal)

[Luca Boccassi]

On Fri, 19 May 2023 at 12:42, Steve McIntyre  wrote:
>
> Guillem Jover wrote:
> >On Thu, 2023-05-18 at 12:01:40 -0700, Steve Langasek wrote:
>
> ...
>
> >> > > * … but NOT on i386.  Because i386 as an architecture is primarily of
> >> > >   interest for running legacy binaries which cannot be rebuilt against 
> >> > > a new
> >> > >   ABI, changing the ABI on i386 would be counterproductive, as 
> >> > > mentioned in
> >> > >   https://wiki.debian.org/ReleaseGoals/64bit-time.
> >>
> >> > Like Russ, I'm also dubious about this, and introduces a special case
> >> > and complexity that does not seem warranted TBH. If this was the case it
> >> > would seem to me it would disallow SOVERSION bumps for example, which we
> >> > have never been concerned with.
> >>
> >> I didn't see anything in Russ's email in this thread that implied he was
> >> dubious of this approach?  He simply has a library he maintains for which 
> >> he
> >> believes binary compatibility is uninteresting.
> >
> >Ah, indeed, just reread the specific paragraph now, sorry Russ! :)
> >
> >> FWIW in Ubuntu where we maintain i386 strictly as an architecture for
> >> compatibility with third-party binaries, we have 1034 source packages that
> >> build Arch: i386 debs.  Not all of those source packages are shared
> >> libraries, but enough of them are that manually updating them to maintain
> >> ABI compatibility on i386 would be a substantial amount of work.  In terms
> >> of overall complexity, I think the scales tip in favor of the toolchain not
> >> defaulting to _TIME_BITS=64 on i386.
> >
> >The problem with obsolete packages is also shared by all other arches,
> >and for those and for local packages the dependency system works for
> >the user and should let them know whether they can upgrade or they
> >would need to remove such packages. For other local FOSS packages
> >they might just be able to rebuild them.
> >
> >Excluding i386 from this transition seems to me will pretty much
> >sentence it, and would also make it rather hard to perform that
> >transition cleanly going forward if people want to keep it alive. And
> >while Debian might eventually remove it from its official ports, we
> >have multiple old ports that are still maintained and used.
> >
> >If the main reason is to support non-free binaries, at least to me
> >that does not seem like a very compelling reason. And people can
> >always use old chroots or similar I guess?
>
> i386 is in a really awkward situation here, I think. Nobody is working
> on it explicitly any more (AFAICS?), but its history as by far the
> most common architecture means that:
>
>  * we still have a (very!) long tail of installations using it
>  * there are *massively* more old binaries available for it, free,
>proprietary *and* locally-built
>
> Moving forwards, we need to make a call on what we want i386 for. I
> was hoping to wait until after bookworm is released to have the meat
> of that discussion, but...
>
> I'm planning on stopping publishing installer images for i386
> soon. Why? We should be strongly encouraging users to move away from
> it as a main architecture. If they're still installing i386 on 64-bit
> hardware, then that's a horrible mistake. If they're still running
> i386 *hardware*, then they should be replacing that hardware with more
> modern, more capable, more *efficient* stuff.
>
> As and when we switch i386 to a secondary status like this (however we
> label it!), then I think we should *only* consider it as a
> compatibility layer for older software. People *could* just use old
> chroots or similar, but the need is likely to be around for a
> while.

+1 for stopping publishing installers for i386, it has been mentioned
many times but it's always worth repeating: electricity costs to keep
running i386 hardware are already way higher than what it costs to buy
a cheap, low-power replacement like a raspberry pi, that also provides
better performance.

Just to clarify, by 'soon' here, do you mean for Bookworm too, or post-Bookworm?

Kind regards,
Luca Boccassi

Re: DEP 17: Improve support for directory aliasing in dpkg

[Luca Boccassi]

On Fri, 26 May 2023 at 08:39, Matthew Vernon  wrote:
>
> Hi,
>
> On 26/05/2023 07:03, Ansgar wrote:
> > On Wed, 2023-05-10 at 14:36 -0700, Russ Allbery wrote:
> >> Ansgar  writes:
> >>> Debian going out of its way to tell derivative users to switch back from
> >>> merged-/usr to split-/usr is the *opposite* of trying to make things as
> >>> smooth for them as possible.
> >>
> >> Yes, I agree with that part and I think I objected to that at the time.
> >> Nonetheless, one bad decision doesn't mean that it is Debian policy that
> >> we don't care about derivatives or their users.  I think we made a mistake
> >> there which is not in alignment with our ideals or our goals.  We should
> >> try to reverse that mistake, not double down on it.
> >
> > My impression is that the tech-ctte disagrees on this point and would
> > not want to reverse the mistake, but double down on it (in your words).
>
> Your impression is incorrect. And assigning motivations to other parties
> during contentious discussions should be done with care if at all.
>
> Consider: it is consistent to believe that it would have been better for
> dpkg not to have had that warning added (quite some time ago now), but
> that by now most derivatives that care will likely have patched it out
> again (mitigating the harm); and if the current work on dpkg is allowed
> to run its course then the warning will probably go away anyway.

That assumes all derivatives track unstable/testing and have taken
action, but it is possible for derivatives to track stable only, and
those would be broken.

Kind regards,
Luca Boccassi

Re: another problem class from /usr-merge [Re: Bug#1036920: systemd: please ship a placeholder in /usr/lib/modules-load.d/]

[Luca Boccassi]

On Tue, 30 May 2023 at 10:53, Helmut Grohne  wrote:
>
> Context for d-devel:
>
> Andreas Beckmann noticed that systemd ships an empty directory
> /usr/lib/modules-load.d. When removing a package that ships a file in
> /lib/modules-load.d (such as multipath-tools), dpkg may in some
> circumstanced delete the empty directory owned by systemd.
>
> On Mon, May 29, 2023 at 07:24:09PM +0100, Luca Boccassi wrote:
> > Given what was discussed:
>
> I think the conclusion is drawn too quickly here.
>
> > - bookworm is in hard freeze
> > - there is no functional impact
>
> In effect, this bug report is an instance of a bug class. I am in the
> process of quantifying its effects, but I do not have useful numbers at
> this time. As an initial gauge, I think it is about 2000 binary packages
> that ship empty directories (which does not imply them to be affected,
> rather this is to be seen as a grossly imprecise upper bound).

Not quite - this bug report is filed against src:systemd, and it's
about precisely one thing - the modules-load.d/ directory, and the
reply you are quoting is talking explicitly about that.

> > - unmerged-usr paths are no longer supported
>
> Then you argue that this bug would affect only unmerged systems, while
> it actually is in reverse. Unmerged systems are unaffected by this bug
> class. The deletion that Andreas describes can only happen due to the
> aliasing introduced by merging. This bug class only affects merged
> systems.

No, this bug report only affects unmerged systems and has no effect on
merged ones, as the actual bug after the analysis and discussion is
that some packages since Bullseye install modules-load.d/ files in the
wrong directory, that nothing actually reads (since Bullseye!),
effectively making them useless, but nobody ever noticed, and I can
only speculate that this could be due to the fact that the vast
majority of systems have been merged and thus there's no difference
(alternatively it could be that such packages have extremely low
popcon, I have not checked). If these packages were used on unmerged
system, these bugs would be very real - the functionality they provide
would be broken.

Now you want to talk about a different issue - that is fine of course,
but it's not what I was talking about.

> In my earlier reply, I also asked Andreas for a practical impact on
> systemd users and suggested lowering the severity of this instance.
> However, there is more to consider. This poses a problem to piuparts and
> thus testing migration. Making piuparts happy is a use case of its own.
> When a mitigation for non-essential adduser broke piuparts (again, I'm
> sorry about that), the release team decided that piuparts is an
> important piece of the release process and therefore the change was to
> be reverted. As a result, apt now depends on adduser in bookworm again.
> To be clear, I fully support the decision that has been made here and
> thank the release team for dealing with resulting issues (e.g. delayed
> migration of other packages). Since the problem we are discussing here
> is quite similar, I argue that this problem class also should be
> considered release critical in general, because it may impact testing
> migration. That being said, IANARM and I therefore leave that judgement
> to others.

/lib/modules-load.d/ being ignored (and thus packages shipping files
there being buggy) has been the case since Bullseye, as already
mentioned. If it was not release critical for Bullseye, why should it
be for Bookworm?

> > - as soon as trixie opens for business we might just canonicalize
> > everything (assuming all the ducks will be in a row)
>
> You make this look like a simple way forward.

The qualifications in that sentence were included to do precisely the opposite.

Kind regards,
Luca Boccassi

Re: another problem class from /usr-merge [Re: Bug#1036920: systemd: please ship a placeholder in /usr/lib/modules-load.d/]

[Luca Boccassi]

On Tue, 30 May 2023 at 14:09, Helmut Grohne  wrote:
>
> Hi Luca,
>
> On Tue, May 30, 2023 at 11:23:07AM +0100, Luca Boccassi wrote:
> > > > - unmerged-usr paths are no longer supported
> > >
> > > Then you argue that this bug would affect only unmerged systems, while
> > > it actually is in reverse. Unmerged systems are unaffected by this bug
> > > class. The deletion that Andreas describes can only happen due to the
> > > aliasing introduced by merging. This bug class only affects merged
> > > systems.
> >
> > No, this bug report only affects unmerged systems and has no effect on
> > merged ones, as the actual bug after the analysis and discussion is
> > that some packages since Bullseye install modules-load.d/ files in the
> > wrong directory, that nothing actually reads (since Bullseye!),
> > effectively making them useless, but nobody ever noticed, and I can
> > only speculate that this could be due to the fact that the vast
> > majority of systems have been merged and thus there's no difference
> > (alternatively it could be that such packages have extremely low
> > popcon, I have not checked). If these packages were used on unmerged
> > system, these bugs would be very real - the functionality they provide
> > would be broken.
>
> Given that we are saying exactly the opposite of each other, it seems
> likely that we are talking about different things (thanks to that kind
> soul pointing it out to me).
>
> As I read your reply, it seems to me that you see the bug in
> multipath-tools and other packages that ship files in
> /lib/modules-load.d as opposed to /usr/lib/modules-load.d. Assuming
> that's your view, what you write very much makes sense - including the
> assertion that it only affects unmerged systems. Do you confirm? If you
> confirm, I'd see what you see as the bug we are talking about as not an
> issue in systemd at all, but as multiple issues in other packages (such
> as multipath-tools) that fail at integrating properly with systemd (when
> unmerged, which is unsupported, so not worth fixing in bookworm). Given
> that the bug at hand is filed against systemd (rather than
> multipath-tools), it did not occur to me earlier that you were having
> this problem in mind.

Yes, that's exactly right - this behaviour was changed a long time ago
(for Bullseye) as mentioned earlier in the thread:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=971282

so it could of course be changed back so that files in /lib are picked
up instead - but as mentioned, I don't think it's worth it, especially
not this late. Packages installed in unmerged Bullseye shipping files
in /lib/modules-load.d have been buggy ever since, and on a merged
system it's just a cosmetic issue. This is (was) a severity serious
bug against src:systemd and that's how I approached it.

> As I understand what Andreas wrote (maybe he can confirm), the problem
> he sees is that /usr/lib/modules-load.d (the directory) disappears when
> removing other packages such as multipath-tools. So it's very much not
> about whether systemd deals with the dropins placed by multipath-tools.
> It's about removal of a package having unintended side-effects (removing
> a directory still owned by systemd). And this very problem, can only be
> experienced on merged /usr. The absence of a directory may not seem like
> a big deal to you and none of us seems convinced that it has a
> practically relevant impact on using systemd, but it very much has an
> impact on piuparts and testing migration and that - to me - is what this
> bug report has been about.
>
> Does that make more sense to you now?

Yes, that makes sense. However I don't think it needs to be a release
blocker? AFAIK there are no load-modules.d/ migrations left to do for
bookworm, and we can make sure piuparts is happy in trixie. Do you see
any reason why this should be a release blocker?

Kind regards,
Luca Boccassi

Re: another problem class from /usr-merge [Re: Bug#1036920: systemd: please ship a placeholder in /usr/lib/modules-load.d/]

[Luca Boccassi]

lution and it seems diversions won't cut it.
> If you can propose any generic workaround or recipe for this situation,
> I'm all ears. The placeholder file sounds ugly, but might work.

I agree, doesn't seem very worrying, and as far as I understand the
observed impact so far is on testing infrastructure, but user
functionality is not impacted, right? If needed, placeholder could be
added, or the testing infrastructure could be taught to ignore them.

Kind regards,
Luca Boccassi

Re: 64-bit time_t transition for 32-bit archs: a proposal

[Luca Boccassi]

On Tue, 6 Jun 2023 at 11:46, Simon McVittie  wrote:
>
> On Tue, 06 Jun 2023 at 09:33:22 +0200, Helmut Grohne wrote:
> > Judging from the conversation, killing i386 quite obviously is desired
> > by some participants, but evidently not by all. How quickly we want to
> > kill it is not obvious to me.
>
> When considering the future of i386, a factor that we need to bear in
> mind is that there are two major use-cases for i386, with requirements
> that sometimes conflict:
>
> 1. i386 as a fully-featured architecture that you can run independently
>on 32-bit x86 systems from roughly the 2000-2010 era
>
> 2. i386 as a multiarch foreign architecture to run legacy binaries on
>modern x86_64 systems
>2a. legacy native Linux i386 binaries
>2b. legacy Windows i386 binaries via Wine (which requires a somewhat
>complete i386 Linux library stack)
>
> For example, Ubuntu has ceased to support i386 for the first use-case,
> and only supports the second use-case. I personally think that Ubuntu
> has made a good pragmatic decision here, which Debian should seriously
> consider adopting.

+1

Given native 32-bit x86 hardware is no longer produced (and has not
been produced for a good while), so it has a de-facto limited life, as
consumer-grade electronics are what they are. Let's say hypothetically
that Trixie is the last full-i386 release, going out in 2025 plus 10
years of LTS+ELTS support that gives a supported OS till 2035, just
shy of the Y2038 deadline and with ~25 years of full support for the
most recent piece of hardware that was produced. And it's not like
those OSes versions are going to evaporate into thin air at that point
- they can still be used. Probably should not be connected to the open
internet and used to browse the web, but realistically already today I
doubt browsing the modern web is a good experience on 32bit hardware
from 2010, I can't imagine it would much better be in 2035...

On the other hand, i386 binaries that can run on x86_64 are
realistically going to be around for much, much longer as you pointed
out, so to me prioritizing this use case seems more prudent and
future-proof between the two alternatives.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Thu, 8 Jun 2023 at 09:46, Raphael Hertzog  wrote:
>
> Hi,
>
> On Wed, 17 May 2023, Helmut Grohne wrote:
> > For completeness sake, there is one more entry in category 3: We can run
> > the dynamic loader from its canonical location explicitly, so we'd
> > modify maintainer scripts to start with:
> >
> > #!/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /usr/bin/sh
> >
> > This would only affect preinst scripts participating in bootstrap and
> > we'd not bother with changing PT_INTERP in the toolchain nor in
> > packages. Unfortunately, this completely breaks the DPKG_ROOT work and
> > with that, I see no viable entries in category 3 for moving forward.
> >
> > Moving on to category 4 feels rather obvious, especially because work
> > has been done there in debootstrap. The approach in debootstrap however
> > is one that I see as a dead end, because it causes us to maintain this
> > code multiple times. It's the number of derivatives times the number of
> > bootstrap tools and that doesn't scale.
> >
> > Category 4 is wider though and we also have other prior art at
> > https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap. In particular,
> > making architecture bootstrap become chrootless would likely be a
> > generic solution to this and other problems.  However, any change needs
> > to propagate to a stable release in all bootstrapping tools.  Therefore
> > we cannot reasonably finish the transition before forky.  This makes
> > category 4 rather unattractive in a short term, but still worth pursuing
> > in a long term.
>
> In the same spirit, I'd like to throw an idea... could we decide that
> base-files is the first package to be configured as part of the bootstrap
> protocol and change base-files maintainer's scripts into statically linked
> executables so that they can work even if we don't have the library loader
> on the ABI-compliant path?
>
> And creating the required symlinks would be done by those (standalone)
> maintainer scripts...
>
> I don't know if we already have some rule/invariant in the configuration
> order of the unpacked packages, but I doubt so.
>
> > Having ruled out categories 3 and 4 maybe category 2 would be good?  We
> > could just ship those symlinks in base-files and be done, right?
> > Unfortunately, we pass -k to tar in debootstrap, so when it extracts
> > base-files and tries to unpack the bin -> usr/bin symlink, it sees that
> > oh no, there already is a symlink at bin (as debootstrap placed it
> > there) and thus fails. So in order to make this work, we also have to
> > modify debootstrap (and thus are in a combination of category 3 and 4).
>
> So when we will have fixed this, and waited for a release cycle, we can
> get rid of the statically compiled maintainer scripts and simply ship the
> symlinks in base-files.

Just a note that we can always change debootstrap via bookworm-p-u,
we've already done so in the past for this, so there's no requirement
to wait an extra release.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Fri, 9 Jun 2023 at 10:53, Raphael Hertzog  wrote:
>
> On Fri, 09 Jun 2023, Marco d'Itri wrote:
> > On Jun 08, Raphael Hertzog  wrote:
> >
> > > In the same spirit, I'd like to throw an idea... could we decide that
> > > base-files is the first package to be configured as part of the bootstrap
> > > protocol and change base-files maintainer's scripts into statically linked
> > > executables so that they can work even if we don't have the library loader
> > > on the ABI-compliant path?
> > It could be even easier: base-files could be unpacked once without
> > running the maintainer scripts and then "reinstalled" again later as
> > usual.
>
> I think you are missing the point here, that only works if the package is
> shipping the symlinks. And the idea is to not do this immediately because
> it breaks debootstrap: if I understood correctly unpacking base-files
> with the symlinks would fail if debootstrap had already pre-created those
> symlinks (due to a -k option that we should get rid of in
> /usr/share/debootstrap/scripts/debian-common).
>
> Hence the special maintainer script to create the required symlinks
> without relying on /bin/sh or any dynamically linked executable.

Yes I think this will necessarily require another round of debootstrap
changes once we've locked in on what we want to do, and go via the
various -p-u queues. I'm pretty sure some buildds will still be stuck
on Buster for example. I've done this last year and I'm happy to do it
again once we have a plan.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems

[Luca Boccassi]

On Sat, 10 Jun 2023 at 09:40, Helmut Grohne  wrote:
> My takeaway here is that while I see the protective diversion as the
> "obviously superior solution", this clearly is not consensus at this
> time. It also means that when rewriting DEP 17, I need to spend quite a
> bit of text on rationale. Thank you.

I would caution to avoid interpreting clarifying questions being asked
as dissent. It's good to ask questions and clarify details about
corner cases, but I wouldn't automatically write them down as
disagreement. At least that's my reading of recent parts of this
thread.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)

[Luca Boccassi]

On Sun, 11 Jun 2023, 14:32 Jeroen Dekkers,  wrote:

> On Fri, 09 Jun 2023 22:14:16 +0200,
> Helmut Grohne wrote:
> > On Fri, Jun 09, 2023 at 02:42:25PM -0500, Richard Laager wrote:
> > > Later, whatever replaces /lib64 with a symlink needs to deal with
> this, but
> > > that's not significantly different than whatever it was going to do
> anyway,
> > > right? Just do this:
> > >
> > > 1. Whatever safety checks are appropriate.
> > > 2. Unless already verified to be identical by #1, hardlink
> > > /lib64/ld-linux-x86-64.so.2 to /usr/lib64/ld-linux-x86-64.so.2. This
> might
> > > be just a particular instance of the more general case of hardlink
> > > everything from /lib64 into /usr/lib64.
> > > 3. Unlink everything from /lib64.
> > > 4. Unlink /lib64.
> > > 5. Symlink /lib64 to /usr/lib64
> >
> > I think we start from the premise that /lib64 already is a symlink and
> > as long as libc6 actually ships /lib64 (even if empty), dpkg won't
> > delete it. What we will not get here is getting rid of the aliasing and
> > we will also be unable to ship /lib64 as a symlink in any data.tar
> > (since that would be a directory vs symlink conflict, which has
> > unpack-order-dependent behaviour, which is bad).
>
> But if all packages in trixie are changed to not ship /lib64 anymore, there
> wouldn't be a conflict in trixie anymore? If we have the following
> situation:
>
> - We change dpkg to never delete /bin, /lib, /lib32, /lib64, /libo32,
> /libx32,
>   and /sbin. We can also change dpkg in bookworm to not do this so that we
> are
>   sure that when upgrading to trixie we have a dpkg that won't delete any
> of
>   these symlinks.


Changing dpkg is a non starter. Even assuming it's doable (it's not) it
would take years of fighting before a single line of code was merged, and
would require finding a new maintainer at a minimum. Are you volunteering
for the job? The file move moratorium would have to remain in place for
2/3/4 releases on top of that while all of this is sorted.

Helmut's idea is good because it's doable in the real world. Are there
theoretical alternatives? Most likely. But let's focus on what's possible
and really doable in the next month or so and get this over with, please.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems

[Luca Boccassi]

On Sun, 11 Jun 2023 at 18:06, Russ Allbery  wrote:
> On the other, related topic, I've also been somewhat confused in this
> discussion why it seems like there's a long-term goal to not have any
> Debian package ship the /bin and /lib symlinks.  I would assume we would
> keep those symlinks forever, and thus will always be shipping them in a
> package from now on (once we get to a place where it's safe to ship them
> in a package).  In the long term, that seems like the most efficient way
> to prevent them from being removed, and also just seems obviously correct
> semantically.

In the long term, there are two camps: those who would like to ship
everything as package content (ie: top level symlinks in data.tar of
some package, probably base-files) and those who would like packages
to exclusively ship files under the vendor trees (/usr and optionally
/etc) and let image builders set up required mount points, symlinks,
and so on.
To be clear, despite being in the latter group, I am absolutely fine
with going with the first option for now to finish the transition, if
it's the best and easiest way to do so, and eventually revisit it
later.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems

[Luca Boccassi]

On Sun, 11 Jun 2023 at 19:07, Russ Allbery  wrote:
>
> Luca Boccassi  writes:
> > On Sun, 11 Jun 2023 at 18:06, Russ Allbery  wrote:
>
> >> On the other, related topic, I've also been somewhat confused in this
> >> discussion why it seems like there's a long-term goal to not have any
> >> Debian package ship the /bin and /lib symlinks.  I would assume we
> >> would keep those symlinks forever, and thus will always be shipping
> >> them in a package from now on (once we get to a place where it's safe
> >> to ship them in a package).  In the long term, that seems like the most
> >> efficient way to prevent them from being removed, and also just seems
> >> obviously correct semantically.
>
> > In the long term, there are two camps: those who would like to ship
> > everything as package content (ie: top level symlinks in data.tar of
> > some package, probably base-files) and those who would like packages
> > to exclusively ship files under the vendor trees (/usr and optionally
> > /etc) and let image builders set up required mount points, symlinks,
> > and so on.
>
> Ah.  Well, let me register my (preliminary, rebuttable) opposition to that
> second strategy in advance to hopefully make it clear well before it
> becomes an issue that there is no current project consensus to go that
> route and we need an actual design discussion (hopefully this time with
> considerably more attention paid to details) before starting down that
> path, if we do.
>
> (Now is not the time for that discussion; please don't try to convince me
> that I'm wrong at the moment.  I'm only asking that people remember that
> this is not something we have all agreed upon.)

No need to worry, as I mentioned there are two camps, as it's clear
and obvious that there is no consensus one way or the other. There's
loads more to do that is more useful and more urgent before it even
gets down to this, as far as I'm concerned.

Kind regards,
Luca Boccassi

Re: proposal: dhcpcd-base as standard DHCP client starting with Trixie

[Luca Boccassi]

On Mon, 19 Jun 2023 at 12:36, Michael Biebl  wrote:
>
> Am 19.06.23 um 12:54 schrieb Martin-Éric Racine:
> > Greetings,
> >
> > Seeing how the ISC DHCP suite has reached EOL upstream, now might be a
> > good time to re-visit Debian's choice of standard DHCP client shipping
> > with priority:important.
> >
> > I hereby propose bin:dhcpcd-base:
> >
> > 1) already supported by ifupdown.
> > 2) dual stack (DHCPv4, Bonjour, RA, DHCPv6 with PD) with privilege 
> > separation.
> > 3) writes both IPv4 and IPv6 name servers to /etc/resolv.conf
> > 4) supports /etc/resolv.conf.head and /etc/resolv.conf.tail
> > 5) a mere inet line in /etc/network/interfaces is sufficient to
> > configure both stacks.
> >
> > While dhcpcd development became somewhat erratic last year due to
> > upstream's health issues, things are seemingly back to normal.
> > Additionally, a new developer has joined and is willing to take over
> > the development should upstream's health deteriorate again.
> >
>
> Why does isc-dhcp-client have priority:important to begin with?
> I don't think users care so much about a dhcp client but rather a
> network configuration system and each network configuration system has
> its own preferred dhcp implementation e.g NetworkManager no longer uses
> isc-dhcp-client by default and systemd-networkd doesn't use
> isc-dhcp-client either.
>
> So maybe simply demoting the priority of isc-dhcp-client to normal is a
> better route.
> ifupdown already recommends isc-dhcp-client and if ifupdown want's to
> use dhcpcd in the future it could change this recommends.

Yes, this sounds like a better approach to me as well.

Kind regards,
Luca Boccassi

Re: proposal: dhcpcd-base as standard DHCP client starting with Trixie

[Luca Boccassi]

On Mon, 19 Jun 2023 at 13:13, Ansgar  wrote:
>
> On Mon, 2023-06-19 at 13:35 +0200, Michael Biebl wrote:
> > Why does isc-dhcp-client have priority:important to begin with?
> > I don't think users care so much about a dhcp client but rather a
> > network configuration system and each network configuration system
> > has its own preferred dhcp implementation e.g NetworkManager no
> > longer uses isc-dhcp-client by default and systemd-networkd doesn't
> > use isc-dhcp-client either.
> >
> > So maybe simply demoting the priority of isc-dhcp-client to normal is
> > a better route.
> > ifupdown already recommends isc-dhcp-client and if ifupdown want's to
> > use dhcpcd in the future it could change this recommends.
>
> The priority question isn't the important one. The real question is:
>
> What network configuration system should users end up with (by
> default)?
>
> I think this should be NetworkManager for desktop environments and I
> personally like systemd-networkd for other environments. In both cases
> these replace both ifupdown and isc-dhcp-client.
>
> I also think that installing both ifupdown and NetworkManager on
> desktop environments is worse than only NM.

The advantage of doing that is that it's what Ubuntu does IIRC, so
there will be extra pooling&sharing of resources to maintain those
setups, and the road should already be paved for it.

Kind regards,
Luca Boccassi

Re: proposal: dhcpcd-base as standard DHCP client starting with Trixie

[Luca Boccassi]

On Mon, 19 Jun 2023 at 20:00, Martin-Éric Racine
 wrote:
>
> On Mon, Jun 19, 2023 at 9:11 PM Santiago Ruano Rincón
>  wrote:
> > El 19/06/23 a las 13:54, Martin-Éric Racine escribió:
> > > Seeing how the ISC DHCP suite has reached EOL upstream, now might be a
> > > good time to re-visit Debian's choice of standard DHCP client shipping
> > > with priority:important.
> > >
> > > I hereby propose bin:dhcpcd-base:
> > >
> > > 1) already supported by ifupdown.
> > > 2) dual stack (DHCPv4, Bonjour, RA, DHCPv6 with PD) with privilege 
> > > separation.
> > > 3) writes both IPv4 and IPv6 name servers to /etc/resolv.conf
> > > 4) supports /etc/resolv.conf.head and /etc/resolv.conf.tail
> > > 5) a mere inet line in /etc/network/interfaces is sufficient to
> > > configure both stacks.
> >
> > I agree that dhcpcd seems the best alternative to isc-dhcp-client for
> > the moment, and I'll make the relevant changes in ifupdown as soon as I
> > can. Josué, any thoughts?
>
> I've never had to do this before, so I wonder if moving packages to
> severity: standard or higher (in this case, important) requires any
> decision from the CTTE or a similar authority, before we proceed?

As mentioned elsewhere in the thread, there shouldn't be any need to
change the priority, adjusting ifupdown's dependencies should be all
that's needed.

Kind regards,
Luca Boccassi

Re: proposal: dhcpcd-base as standard DHCP client starting with Trixie

[Luca Boccassi]

On Mon, 19 Jun 2023 at 18:21, Philipp Kern  wrote:
>
> On 2023-06-19 14:37, Luca Boccassi wrote:
> > The advantage of doing that is that it's what Ubuntu does IIRC, so
> > there will be extra pooling&sharing of resources to maintain those
> > setups, and the road should already be paved for it.
>
> I am not sure if I have seen this play out in practice[1].
> Ubuntu^WCanonical has been doing its own development in this space as
> well with netplan. Ubuntu will continue to do its own fixes to glue
> things together.
>
> Kind regards
> Philipp Kern
>
> [1] With notable exceptions like doko maintaining the toolchain - and
> I'm sure I'm not crediting everyone. But that's also explicit package
> maintainership.

I've been working for a long time with many Canonical engineers,
happily and fruitfully, to the benefit of both Debian and Ubuntu, so
my experience is quite different. This includes the developers working
on src:systemd.

Kind regards,
Luca Boccassi

Re: proposal: dhcpcd-base as standard DHCP client starting with Trixie

[Luca Boccassi]

On Tue, 20 Jun 2023 at 11:42, Simon McVittie  wrote:
>
> On Tue, 20 Jun 2023 at 05:03:19 -0400, nick black wrote:
> > Simon McVittie left as an exercise for the reader:
> > > At the moment I believe the status quo for d-i is that networking is
> > > managed by NetworkManager if a desktop task happens to have pulled it in,
> > > or ifupdown otherwise? And that seems reasonable (although I personally
> > > prefer to set up systemd-networkd on servers).
> >
> > i don't wish to start an argument, but just to ask: everyone has
> > recommended NetworkManager for workstations. i've been running
> > systemd-networkd on everything (servers, laptops, and
> > workstations) for several years now, usually in conjunction with
> > dnsmasq and wpa_supplicant, and it's been pretty great. what
> > does NetworkManager offer that makes it superior to
> > systemd-networkd on the desktop (which i'm interpreting to mean
> > "for interactive use")?
>
> Ubiquitous user interfaces, mostly. Our default for laptops and other
> portable computers needs to be something that lets a non-technical user
> of GNOME/Plasma/etc. join a wireless network, without learning how to
> write configuration files and operate sudo, and in practice that means
> the various desktop environments' UIs for NM (or something analogous
> like connman or wicd, but NM seems to be by far the best-supported choice
> out of those).
>
> I don't know enough about implementation details of NM and
> systemd-networkd to know whether the API design of NM is more suitable
> for that purpose, or whether the various UIs for NM and the absence of
> UIs for systemd-networkd is just inertia; but in practice the network
> configuration service that has first-class support in most of our desktop
> environments (in particular GNOME and Plasma) is NM.
>
> I was using "desktop" in the sense of task-gnome-desktop and friends, more
> than as a class of hardware. Laptops and other portable computers are the
> main thing that really needs easily user-configurable networking.
> I think it makes sense for desktop/workstation hardware to be treated like
> an oddly-shaped laptop by default, which means it gets the benefit of the
> wider testing that goes into NM and its various user interfaces, rather
> than having laptops and desktops behave differently for reasons that are
> unlikely to be obvious to a new user.
>
> Some users of desktop/workstation hardware strongly prefer to use a more
> "static" network configuration service like systemd-networkd or ifupdown
> so that they can rely on having the network setup not change under
> them, particularly if they're using services like NFS filesystems or
> LDAP authentication. That's an entirely reasonable thing to want to do,
> but IMO this is an example of the design principle that the choice that
> is better for non-technical users can make a better default, because
> technically adept users who know they have particular requirements can
> easily switch to what we might characterize as "server" infrastructure,
> but non-technical users can't easily switch in the opposite direction
> (or even know that they might want to).
>
> A secondary benefit of NM is that it works on non-systemd-booted systems,
> whereas systemd-networkd isn't designed for that use. I'm personally
> happy with systemd as pid 1, but some people consider requiring systemd
> as pid 1 to be a deal-breaker, and if NM is a good candidate for being
> our default *anyway* then we might as well get that secondary benefit too.

Yes a fully working and well-integrated GUI is the most important
thing for the default (as in, desktop tasklet default), power users
can always switch to something else. NetworkManager is clearly ahead
of the alternatives in that regard.

Kind regards,
Luca Boccassi

Re: Default network configuration system (was Re: proposal: dhcpcd-base as standard DHCP client starting with Trixie)

[Luca Boccassi]

"netplan 
> status".
>
> Cheers,
> Lukas
>
> [1] https://tracker.debian.org/pkg/netplan.io
> [2] 
> https://hamburg-2023.mini.debconf.org/talks/5-network-configuration-on-debian-systems/

Yeah I think it would be nice to do some experiments there, at least
starting with networkd and eventually considering netplan as an
additional step on top - thanks for volunteering :-P

Ultimately it seems to me that the defaults should be driven by
tasklets rather than priority - ie, make ifupdown and dhcpd-base
optional, have ifupdown depend on dhcpd-base so that if a user wants
to use that stack they can just pull in ifupdown and it works as
expected, and have the desktop task pull in NetworkManager. Then we
can decide what the default stack for servers should be after some
experimenting.

Kind regards,
Luca Boccassi

Re: Policy consensus on transition when removing initscripts.

[Luca Boccassi]

On Sun, 25 Jun 2023 at 15:21, Mark Hindley  wrote:
>  "Init systems other than ``systemd`` should allow providing the same
>  functionality as appropriate for each system, for example managing the
>  directories from the init script shipped by the package."
>
> This creates an inconsistency whereby non-systemd inits are required to 
> provide
> functionality in their initscript, but that initscript is not required to be
> present.

This wasn't intended as a hard rule, I used "should" and "for example"
to try and make that clear. That text has now been made more generic
after a suggestion:

-directories from the init script shipped by the package.
+directories from the appropriate service startup configuration.

which should hopefully clarify that nothing is required to be
implemented from scripts or packages themselves, but it is merely a
hint to developers and maintainers of alternatives that such
functionality should be supported in whichever way is most appropriate
for their systems.

Does this change help to clarify the intentions?

Kind regard,
Luca Boccassi

Re: MBF: packages shipping init scripts without corresponding systemd units

[Luca Boccassi]

On Sun, 25 Jun 2023 at 22:29, Luca Boccassi  wrote:
>
> Hi,
>
> According to Lintian there are 314 packages shipping init scripts
> without a corresponding systemd unit:
>
> https://lintian.debian.org/tags/missing-systemd-service-for-init.d-script
>
> They currently work because there is still a transitional unit
> generator that creates a unit on-the-fly on boot. This was always
> intended as a temporary stop-gap, and is technically vastly inferior to
> a native unit, as for example it cannot tell the difference between a
> one-shot and a long-running service, and cannot enable any hardening or
> sandboxing options.
>
> Now the generator is also on the way to be deprecated and removed. It's
> been there for a decade, which is enough time to complete the
> transition, and will likely be removed before Trixie ships.
>
> Therefore I filed a bug against all affected packages, provided a patch
> for policy:
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039102
>
> and a patch for Lintian to bump the severity from warning to error:
>
> https://salsa.debian.org/lintian/lintian/-/merge_requests/407
>
> It's possible that there will be a good chunk of false positives, as
> often new units added don't have a name that matches exactly the old
> init script name, in which case it's fine to add an override and close
> the bug.

It would probably make things easier if I typed the destination
address correctly.

Re: MBF: packages shipping init scripts without corresponding systemd units

[Luca Boccassi]

On Mon, 26 Jun 2023 at 05:21, Paul Wise  wrote:
>
> On Sun, 2023-06-25 at 22:31 +0100, Luca Boccassi wrote:
>
> > Now the generator is also on the way to be deprecated and removed.
> ...
> > Therefore I filed a bug against all affected packages ...
>
> Will the generator be available separately for people who have packages
> not in Debian that still only have init scripts, or for people who have
> local init scripts installed manually or with config management tools?

It will not, it's going to be removed upstream and hence from
src:systemd. Exact date/version is not planned yet, but it will be
before Trixie's dev cycle ends. It will be written down in the release
notes when the time comes.

Kind regards,
LUca Boccassi

Re: Policy consensus on transition when removing initscripts.

[Luca Boccassi]

On Tue, 27 Jun 2023 at 04:10, Paul Wise  wrote:
>
> On Mon, 2023-06-26 at 20:04 -0400, nick black wrote:
>
> > furthermore, this is only true when procfs is mounted with a
> > nonzero hidepid, right?
>
> I note that systemd does not support non-zero hidepid, so
> procfs hidepid will always be off on systemd based systems:
>
> https://github.com/systemd/systemd/issues/12955
>
> At least until Linux offers a way for systemd to have access
> to /proc but other programs to not have access to it.

That has been implemented a long time ago, services can set
ProtectProc= so that processes run with hidepid:

https://freedesktop.org/software/systemd/man/systemd.exec.html#ProtectProc=

Kind regards,
Luca Boccassi

Re: Policy consensus on transition when removing initscripts.

[Luca Boccassi]

On Tue, 27 Jun 2023 at 01:04, nick black  wrote:
>
> Simon McVittie left as an exercise for the reader:
> > started as root and dropped privileges to some other uid, that permanently
> > restricts its ability to read information out of its own /proc, which is
> > not always desirable. If the daemon starts up unprivileged, then it can
>
> i assume by "its own /proc" you mean /proc/getpid()? i don't see
> how this is different from any other resource one might need
> consider acquiring prior to dropping privileges. if i want to
> open a privileged port, i'd better do that before i change my
> user (or otherwise yield CAP_NET_BIND_SERVICE).
>
> furthermore, this is only true when procfs is mounted with a
> nonzero hidepid, right? all my /proc/PID directories are 0755,
> with contents likewise generally world-readable. hidepid=off is
> the default according to
> https://www.kernel.org/doc/html/latest/filesystems/proc.html.

Resources are dynamic, and come and go - for example, file
descriptors. The 'run as root and drop privileges' pattern might have
been fine in the past century, but in 2023 we have much, much better
patterns to secure running services. And using hidepid (ie,
ProtectProc) and other sandboxing on services is one such pattern.

Kind regards,
Luca Boccassi

Re: booststrapping /usr-merged systems

[Luca Boccassi]

On Sun, 11 Jun 2023 at 22:17, Timo Röhling  wrote:
>
> * Luca Boccassi  [2023-06-10 19:54]:
> >I would caution to avoid interpreting clarifying questions being asked
> >as dissent. It's good to ask questions and clarify details about
> >corner cases, but I wouldn't automatically write them down as
> >disagreement. At least that's my reading of recent parts of this
> >thread.
>
> This is also my understanding. And for the record, I want to
> emphasize that I am very much in favor of the plan that Helmut came
> up with, for a number of reasons:
>
> [Full disclosure: I had a few in-person discussions with Helmut in
> Hamburg last month, so I am probably somewhat biased by now.]
>
>
> 1. Helmut has shown experimentally that his transition plan can
> work. There are always unknown unknowns, of course, but at the very
> least, we do not have to break any use-cases intentionally.
>
> 2. The transition will leave us in a well-defined state post-trixie
> without the need to add (and continue to maintain) any clutches
> (or "special cases") for dpkg.
>
> 3. Almost all problematic cases can be dealt with by some black
> magic in a single usrmerge-support package. It is not pretty, but it
> will get the job done; a bunch of trickery to make dpkg do the Right
> Thing despite its incomplete knowledge of aliased paths.
>
> 4. We will be able detect the few cases where the Right Thing does
> not happen transparently, and we can even give advance warning to
> affected package maintainers what they should and should not do. If
> the maintainers of those packages pre-upload their transitioned
> packages to experimental for some automated tests and verification,
> we can avoid any breakage in unstable and testing.
>
>
> Of course, you do not have to take my word for any of this. I am a
> big fan of Helmut's approach with experimental verification and
> data-driven discovery. Have a look at his published test scripts and
> try to poke holes in them. The more people do this, the more
> confidence we can have that this might actually work after all.

Hi Helmut,

Any update on this topic? I believe you were working on a write-up,
how's that going?

Kind regards,
Luca Boccassi

Re: Policy consensus on transition when removing initscripts.

[Luca Boccassi]

On Wed, 28 Jun 2023, 06:31 Paul Wise,  wrote:

> On Tue, 2023-06-27 at 09:36 +0100, Luca Boccassi wrote:
>
> > That has been implemented a long time ago, services can set
> > ProtectProc= so that processes run with hidepid:
> >
> >
> https://freedesktop.org/software/systemd/man/systemd.exec.html#ProtectProc=
>
> Thats opt-in and for services only, there are folks who want to run
> their entire system with hidepid=2, including root/user command-line
> and graphical sessions, systemd doesn't support this setup.
>

On the global proc instance yes that cannot possibly work, as the kernel
doesn't make the required information to do process management available.
But with the per service setting you can set it on the graphical session
(eg gdm.service), however it's likely that it won't cover everything, and
also unrelated bits might break.

>

Re: Developer Workload and Sustainability

[Luca Boccassi]

On Wed, 28 Jun 2023 at 17:20, Ansgar  wrote:
>
> Hi Simon,
>
> On Thu, 2023-06-29 at 00:32 +0900, Simon Richter wrote:
> > According to Policy as currently published, systemd units are
> > encouraged, and init scripts are mandatory.
>
> Please stop lying:
>
> +---
> | Packages that include system services should include systemd service
> | units to start or stop those services. [...]
> |
> | If the package does not include a service unit (if, for example, no
> | one has yet written one), including an init script, as described
> | below, to start the service is encouraged.
> |
> | Packages including a service unit may optionally include an init
> | script to support other init systems.
> +---[ Policy 9.3.1 ]
>
> And no, this isn't exactly new. Except apparently for you.
>
> The real exhausting thing is people lying, FUD, spreading conspiracy
> theories (ominous commercial sponsors (Rothschilds? Soros?)), endless
> revisiting of decisions (should we prepare to revert usrmerge because
> the attention of ominous commercial sponsors might shift elsewhere?),
> claiming systemd is rot/cancer/an infection/the Windows registry and so
> on.
>
> I agree that this isn't a technical issue though.
>
>
> > That's the thing: few people want init scripts. I don't want init
> > scripts either.
>
> This very thread only exists because people want init scripts.  I agree
> that this is very few people though.
>
> > What I want is an init system that can be maintained inside a community
> > project like Debian without burning out people and endangering the long
> > term viability of the project.
>
> And claiming this isn't possible with systemd is one thing: FUD.
>
> > That is where systemd fails us as a community project, because the
> > environment in which most of development is happening is not
> > hospitable to community building efforts, and the complexity of the
> > project constitutes a high barrier to entry, which acts as a further
> > selection filter for contributors.
>
> More FUD.

Let's look at some community numbers for two init system projects in
question, we like to look at hard data after all, don't we. To spicy
it up, without mentioning which is which:

$ git shortlog -s | cut -c8- | wc -l
2318

$ git shortlog -s | cut -c8- | wc -l
7

Just from this alone, it certainly seems true that one of those is a
project that "fails as a community project" and that is "not
hospitable to community building efforts". Anyone want to guess which
is which?

Kind regards,
Luca Boccassi

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Luca Boccassi]

#x27;
environment, in the sense that there is some expectation that it might
fail and the runner is expected to deal with it. Moving the risk to
the upgrade phase of every single Debian instance in existence seems
inherently more risky to me.

> Option #4a:
>
> The finalization of the /usr-merge transition shall not rely on
> changes to dpkg, because such changes cannot be relied upon during
> the upgrade to trixie.
>
> This is denying M1 and M3 from DEP17.

+1 totally opposed to have _any_ dpkg changes in the way of any resolution

> Option #5a:
>
> The paths used to interface with update-alternatives remain
> unchanged.
>
> This is M13 from DEP17.

Yeah, let's punt it for now, and deal with the big items first.

> I also hope that this mail results in detailed disagreements that I can
> use to refine DEP17 and to base further research on.

I hate to ask, given you have already put a lot of work on the linked
document, but a very rough ballpark estimate in how much work it would
be to implement each solution (where not available yet) would be very
useful for making decisions, I think. If it's too much effort feel
free to just tell me to go chop some wood.

Thanks for the update!

Kind regards,
Luca Boccassi

Re: Second take at DEP17 - consensus call on /usr-merge matters

[Luca Boccassi]

On Thu, 29 Jun 2023 at 11:53, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi,
>
> mmdebstrap author here. This is the other bootstrapping tool which is 
> currently
> sitting at ~17% of the popcon value of debootstrap.
>
> Quoting Helmut Grohne (2023-06-28 21:37:44)
> > Once that is settled, the next big question is how to handle bootstrapping.
> > We had a number of people arguing in favour of changing the bootstrap
> > protocol. Such changes can be classified into generic changes and
> > release-dependent changes. A release-dependent change enhances bootstrapping
> > tools with knowledge of available releases and adapts behaviour in
> > release-specific ways that are encoded into the bootstrapping tool. As it
> > stands, the only bootstrapping tool that has been enhanced in this way is
> > debootstrap and that support is limited to Debian and two derivatives. The
> > category of generic changes includes imposing an ordering on initial unpacks
> > (e.g. base-files first). Others are in favour of not changing the bootstrap
> > protocol. In that view, some data.tar will have to ship the symbolic links
> > and all other essential packages need to have their files canonicalized.
> >
> > Among these options, the first has a working prototype (debootstrap),
> > but it is unclear how that extends to use of snapshot.d.o and how to
> > make it work with debootsnap and debbisect as those tend to use a suite
> > name rather than a codename. The last option has a prototype and relies
> > on uploading a number of essential packages in a short window of time.
> > (What could possibly go wrong?)
> >
> > It is not clear to me how we can get to a consensus on these, so the
> > best I can do here is summarize options.
> >
> > Option #3a:
> >
> > The bootstrap protocol shall be changed to contain a task for
> > merging /usr as is done in debootstrap in a release-dependent way.
> >
> > This is an instance of M16 from DEP17.
> >
> > Option #3b:
> >
> > The bootstrap protocol shall be changed in unspecified ways to
> > support the /usr-merged systems in a way that does not depend on
> > matching the codename or suitename.
> >
> > This is an instance of M16 from DEP17.
> >
> > Option #3c:
> >
> > The bootstrap protocol shall remain unchanged. Therefore, all
> > essential packages need to move their files out of aliased locations
> > and the aliasing symlinks are to be installed from a data.tar of a
> > binary package such as base-files.

<...>

> I think this now comes down to what we as a project care about. Which 
> use-cases
> are important to us? Which properties do we want to preserve?
>
> What do you think?

Essentially, this boils down to risks vs benefits. The risk of going
3c is that every single Debian installation in existence breaks in
some interesting ways, as fixing the bootstrapping corner case is
delegated to the package upgrade workflow. The sole benefit is that
one of the two bootstrapping tools in widespread use keeps its
internal code a bit 'cleaner' from the point of view of some
technically unnecessary and self-imposed design constraints (yes
there's 2 more tools as pointed out yesterday, but they appear to be
at least under-maintained judging from tracker.d.o).

I don't see how it's worth the risk. This is essentially a problem in
the bootstrapping tools, so solving it in the bootstrapping tools is
not only the safest approach - worst case scenario, creating a new sid
chroot might not work for a couple days, not a big deal given it
happens all the time for various reasons as we've seen this week -
it's also the right approach.

Kind regards,
Luca Boccassi

Re: Replaces without Breaks or Conflicts harmful? [was: Re: Policy consensus on transition when removing initscripts.]

[Luca Boccassi]

On Thu, 29 Jun 2023 at 13:34, Helmut Grohne  wrote:
>
> Hi Bas,
>
> On Thu, Jun 29, 2023 at 08:19:51AM +0200, Sebastiaan Couwenberg wrote:
> > On 6/28/23 21:49, Helmut Grohne wrote:
> > > Debian GIS Project 
> > > postgis
> > > qgis
> >
> > Why is postgis on this list?
> >
> >  $ grep -c Replaces debian/control*
> >  debian/control:0
> >  debian/control.in:0
>
> Thanks for asking. You identified another source of false positives that
> slipped my mind when doing the analysis. The underlying data source did
> not use unstable, but every suite from bullseye to experimental
> including -security and -backports. As it happens, bookworm's
> postgresql-15-postgis-3-scripts has versioned Replaces that are not
> matched with Breaks or Conflicts. I don't think we are going to fix that
> in bookworm and you've fixed it in unstable. So yeah, this list has more
> false positives than originally assumed.

In case it's useful, src:dpdk is also a false positive, I suspect
because the versions in the breaks vs replaces are slightly different
- probably clerical mistakes, like a missing '~'.

> I could improve the numbers, but to me the numbers I've given being a
> tight upper bound seems good enough and lintian.debian.org will give us
> precise and current numbers once my patch is merged. Does that seem
> sensible to you as well?

Sadly, as I found out recently for the scripts mbf, lintian.d.o is
borken and has ~2 years old stale data. We should probably consider
taking it down, given it appears fully working and can be queried, but
just returns stale data with no indication that it is stale on the
face of it, without manual checks.

Kind regards,
Luca Boccassi