Re: Is there still a point in installing libgcrypt to /lib instead of /usr/lib

[Marco d'Itri]

On Feb 16, Simon McVittie  wrote:

> To be clear, what Guillem means by "a proper /usr-merged migration"
> here is changing individual library packages, so that the path to their
Everything I suppose, not just libraries.

> I think we have consensus that consolidating all static OS files into /usr
> (removing the distinction between /usr and the static parts of the root
> filesystem) is the route that Debian is taking. I think we do not have
> consensus on how that is to be achieved.
Do we really have a consensus? Is everybody persuaded now that there is 
no point in continuing to support non-merged systems?

> I would be grateful if people who advocate transitioning individual
> packages, and people who consider the approach taken by usrmerge and
> debootstrap to be sufficient, could refer to their preferred route in a
> way that makes it clear which one they are advocating. Saying we should
I never considered usrmerge (the package) to be the complete solution 
but only a transition method.
I fully support plans to start moving everything from /bin /sbin
/lib to /usr, but I did not want to wait the many years that this will 
take to have a real merged-/usr system.
Libraries are easy to move since the linker will find them no matter 
where they are installed, but moving the binaries too will require 
merging older systems. And to do this we need some work in the usrmerge 
package. Short summary: it used to work very well on a live system, but 
nowadays systemd creates a lot of bind mounts so it should be run from 
the initramfs. But I have not been able to actually do this: if anybody 
wants to have a look at it there is a branch in the repository.
I see no point in only moving libraries.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Y2038 - best way forward in Debian?

[Marco d'Itri]

On Feb 17, Wouter Verhelst  wrote:

> - Figure out a way for 32-bit binary-only programs to keep working when
>   they touch a time_t beyond 2038.
I am sure that around 2035 "time namespaces" or something similar will 
be implemented to solve this.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Y2038 - best way forward in Debian?

[Simon Richter]

Hi,

On Tue, Feb 18, 2020 at 12:25:48PM +0100, Marco d'Itri wrote:

> > - Figure out a way for 32-bit binary-only programs to keep working when
> >   they touch a time_t beyond 2038.

> I am sure that around 2035 "time namespaces" or something similar will 
> be implemented to solve this.

Hopefully a bit earlier. I have a few pieces of software that only build in
virtual machines with a timezone offset to UTC of -10 years, and I'd really
like to move them to containers or light them on fire.

   Simon

Bug#951585: ITP: golang-github-jaytaylor-html2text -- Golang HTML to plaintext conversion library

[Alois Micard]

Package: wnpp
Severity: wishlist
Owner: Alois Micard 

* Package name: golang-github-jaytaylor-html2text
  Version : 0.0~git20190408.01ec452-1
  Upstream Author : J. Elliot Taylor
* URL : https://github.com/jaytaylor/html2text
* License : Expat
  Programming Lang: Go
  Description : Golang HTML to plaintext conversion library


This package is needed for advancedlogic/GoOse

Thanks in advance.

-
Aloïs Micard 

GPG: rsa4096/A5999EBDFC063F1F

Bug#951586: ITP: isodatetime -- Python ISO 8601 date time parser and data model/manipulation utilities

[Alastair McKinstry]

Package: wnpp
Severity: wishlist
Owner: Alastair McKinstry 

* Package name: isodatetime
  Version : 2.0.1
  Upstream Author : Oliver Sanders
* URL : https://github.com/metomi/isodatetime
* License : LGPL 3.0
  Programming Lang: Python
  Description : Python ISO 8601 date time parser and data 
model/manipulation utilities


This is a dependency of the latest version of 'cylc', already in Debian.

Python ISO8601 (2004) full-specification parser and data model/manipulation 
utilities. Intended to be used in a similar way to Python's datetime module.

ISO8601 (2004) is an international standard for writing down date/time 
information.

It is the correct, internationally-friendly, computer-sortable way to 
numerically represent date/time information.

I intend to maintain this with the Python packaging team.

Re: Is there still a point in installing libgcrypt to /lib instead of /usr/lib

[Sam Hartman]

> "Marco" == Marco d'Itri  writes:
>> I think we have consensus that consolidating all static OS files
>> into /usr (removing the distinction between /usr and the static
>> parts of the root filesystem) is the route that Debian is
>> taking. I think we do not have consensus on how that is to be
>> achieved.
Marco> Do we really have a consensus? Is everybody persuaded now
Marco> that there is no point in continuing to support non-merged
Marco> systems?

That is not my understanding of the TC decision no.
My understanding without rereading a complex decision is that for now
we're still supporting both.
But that it's certainly fine if packages move things to /usr if they do
so in a way that avoids breaking things.

How to undo a merged-user installation?

[Svante Signell]

Hello,

I recently installed Debian/bullseye/sid in a VM from a snapshot. After running
that image I realized the I got a merged-user installation.

As for now I have:
bin -> usr/bin
lib -> usr/lib
lib32 -> usr/lib32
lib64 -> usr/lib64
libx32 -> usr/libx32
sbin -> usr/sbin

Is there some way to achieve a non-merged-user system from the current
situation? 

The Debian Installer does not seem to have an option for a non-merged-user
installation. Is that true?

Thank you for your time!

Re: Announcing miniDebConf Montreal 2020 -- August 6th to August 9th 2020

[Ian Jackson]

Roberto C. Sánchez writes ("Re: Announcing miniDebConf Montreal 2020 -- August 
6th to August 9th 2020"):
> On Sat, Feb 15, 2020 at 12:05:29AM -0500, Jerome Charaoui wrote:
> > Following the announcement of the DebConf20 location, our desire to
> > participate became incompatible with our commitment toward the Boycott,
> > Divestment and Sanctions (BDS) campaign launched by Palestinian civil
> > society in 2005. Hence, many active Montreal-based Debian developpers,
> > along with a number of other Debian developpers, have decided not to
> > travel to Israel in August 2020 for DebConf20.
> 
> Would it be possible to not constantly air our personal political
> opinions and grievances on Debian lists?  Especially as part of
> something that goes to an -announce list.

The choice of venue for Debconf is a political act.  In this
particular case, it is highly political.  It is political in a way
that the people who advocated for this venue, and those who chose this
venue, surely recognised.

Pretending that only the dissenting opinions are political, and then
asking for "political" opinions not to be aired, reframes the debate
in such a way that only the status quo can even be expressed.

It was IMO completely appropriate for the Montreal team to make their
position, and their actual motives, clear.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: How to undo a merged-user installation?

[jnqnfe]

On Tue, 2020-02-18 at 17:58 +0100, Svante Signell wrote:
> Hello,
> 
> I recently installed Debian/bullseye/sid in a VM from a snapshot.
> After running
> that image I realized the I got a merged-user installation.
> 
> As for now I have:
> bin -> usr/bin
> lib -> usr/lib
> lib32 -> usr/lib32
> lib64 -> usr/lib64
> libx32 -> usr/libx32
> sbin -> usr/sbin
> 
> Is there some way to achieve a non-merged-user system from the
> current
> situation? 
> 
> The Debian Installer does not seem to have an option for a non-
> merged-user
> installation. Is that true?
> 
> Thank you for your time!

As I understand it (I somehow only became aware of it days ago) a move
to merged usr has been a move planned for Debian since ~2014 and all
more recent installers implement it by default. I would not expect any
available switch in the installer to avoid it (why add complexity to
support the older layout that surely no-one needs?).

I have to ask, why are you determined to spend effort reversing the
change?

While merging is of course trivial, un-merging is most certainly not.
The easiest methods that occur to me would be to either:
 a) if you have access to an older version of the installer, do a fresh
install with that (then of course upgrade).
 b) make sure the usrmerge package is not installed; manually switch
the symlinks back to actual directories; then ask apt/aptitude to do a
reinstall of every package on your system.

Of course I cannot give you any guarantee that the above procedure will
actually be successful. Proceed at your own risk. :)

Re: How to undo a merged-user installation?

[jnqnfe]

On Tue, 2020-02-18 at 17:35 +, jnq...@gmail.com wrote:
> On Tue, 2020-02-18 at 17:58 +0100, Svante Signell wrote:
> > Hello,
> > 
> > I recently installed Debian/bullseye/sid in a VM from a snapshot.
> > After running
> > that image I realized the I got a merged-user installation.
> > 
> > As for now I have:
> > bin -> usr/bin
> > lib -> usr/lib
> > lib32 -> usr/lib32
> > lib64 -> usr/lib64
> > libx32 -> usr/libx32
> > sbin -> usr/sbin
> > 
> > Is there some way to achieve a non-merged-user system from the
> > current
> > situation? 
> > 
> > The Debian Installer does not seem to have an option for a non-
> > merged-user
> > installation. Is that true?
> > 
> > Thank you for your time!
> 
> As I understand it (I somehow only became aware of it days ago) a
> move
> to merged usr has been a move planned for Debian since ~2014 and all
> more recent installers implement it by default. I would not expect
> any
> available switch in the installer to avoid it (why add complexity to
> support the older layout that surely no-one needs?).
> 
> I have to ask, why are you determined to spend effort reversing the
> change?
> 
> While merging is of course trivial, un-merging is most certainly not.
> The easiest methods that occur to me would be to either:
>  a) if you have access to an older version of the installer, do a
> fresh
> install with that (then of course upgrade).
>  b) make sure the usrmerge package is not installed; manually switch
> the symlinks back to actual directories; then ask apt/aptitude to do
> a
> reinstall of every package on your system.
> 
> Of course I cannot give you any guarantee that the above procedure
> will
> actually be successful. Proceed at your own risk. :)

edit: what on earth am I thinking, solution B isn't going to be
sufficient unless DPKG is brilliant enough to remove things from the
old locations; you're ineviatably going to end up still with the merged
copies of things floating about in the merged-to directories...

to solve that, you cannot exactly just delete the directories before
asking apt to reinstall because then you'll likely have broken apt.
perhaps after an all-package reinstall you can then purge all files
from the merged-to directories that exist in the now unmerged
directories (then do a second round of all-packages reinstall, or find
a means of checking for packages with missing files to reinstall, to be
safe).

as I say, non-trivial and proceed at your own risk...

Re: Y2038 - best way forward in Debian?

[Russ Allbery]

Simon Richter  writes:
> On Tue, Feb 18, 2020 at 12:25:48PM +0100, Marco d'Itri wrote:

>> I am sure that around 2035 "time namespaces" or something similar will 
>> be implemented to solve this.

> Hopefully a bit earlier. I have a few pieces of software that only build in
> virtual machines with a timezone offset to UTC of -10 years, and I'd really
> like to move them to containers or light them on fire.

Time namespaces were merged for Linux 5.6.

https://lwn.net/Articles/766089/
https://lwn.net/Articles/810780/

-- 
Russ Allbery (r...@debian.org)  

Re: Y2038 - best way forward in Debian?

[Ben Hutchings]

On Tue, 2020-02-18 at 12:25 +0100, Marco d'Itri wrote:
> On Feb 17, Wouter Verhelst  wrote:
> 
> > - Figure out a way for 32-bit binary-only programs to keep working when
> >   they touch a time_t beyond 2038.
> I am sure that around 2035 "time namespaces" or something similar will 
> be implemented to solve this.

Linux 5.6 introduces time namespaces - though currently they only allow
offsets to monotonic clocks (so containers can have monotonic time when
migrated), not CLOCK_REALTIME.

Ben.

-- 
Ben Hutchings
Tomorrow will be cancelled due to lack of interest.




signature.asc
Description: This is a digitally signed message part

Re: How to undo a merged-user installation?

[Guillem Jover]

On Tue, 2020-02-18 at 17:58:30 +0100, Svante Signell wrote:
> I recently installed Debian/bullseye/sid in a VM from a snapshot. After
> running that image I realized the I got a merged-user installation.
> 
> As for now I have:
> bin -> usr/bin
> lib -> usr/lib
> lib32 -> usr/lib32
> lib64 -> usr/lib64
> libx32 -> usr/libx32
> sbin -> usr/sbin
> 
> Is there some way to achieve a non-merged-user system from the current
> situation? 

Unfortunately you'll have to consider the damage from the default
installer layout irreversible. You could, while not running the system,
remove the symlinks, then move things around based on the pathnames known
to dpkg, but that would still miss all the mess with compat symlinks that
need to be generated by maintainer scripts due to having to support the
merged-/usr-via-symlinks hack. :/

My recommendation is to reinstall from scratch:

 * If using d-i, you'll need to install in expert mode, and proceed just
   before the “install the base system” (or similarly named) step,
   execute a shell and edit «/var/lib/dpkg/info/bootstrap-base.postinst»
   and add «--no-merged-usr» to the «run-debootstrap» call. Then exit the
   shell and proceed with the installation. I've done this recently
   and it works fine.

 * If you are using raw debootstrap then just pass that option. Or
   just switch to use mmdebstrap which does not have a broken default,
   and is way way faster anyway.

 * Otherwise you'll need to fix whatever is generating those snapshots.

> The Debian Installer does not seem to have an option for a non-merged-user
> installation. Is that true?

That would be #923091 with a patch provided by Colin Watson
,
which unfortunately missed the buster release.

Thanks,
Guillem

Bug#951609: ITP: azure-data-lake-store-python -- Azure Data Lake Store Filesystem Library for Python

[Luca Boccassi]

Package: wnpp
Severity: wishlist
Owner: "Luca Boccassi" 
X-Debbugs-CC: debian-devel@lists.debian.org
Control: block 949763 by -1

* Package name: azure-data-lake-store-python
  Version : 0.0.48
  Upstream Author : Microsoft Corporation
* URL : https://github.com/Azure/azure-data-lake-store-python
* License : MIT
* Programming Lang: Python
* Description : Azure Data Lake Store Filesystem Library for Python

A pure-python interface to the Azure Data-lake Storage system,
providing pythonic file-system and file objects, seamless transition
between Windows and POSIX remote paths, high-performance up- and down-
loader.

It is required as a new dependency of the new version of python-azure.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: Is there still a point in installing libgcrypt to /lib instead of /usr/lib

[Simon McVittie]

On Tue, 18 Feb 2020 at 12:44:18 +0100, Marco d'Itri wrote:
> On Feb 16, Simon McVittie  wrote:
> > I think we have consensus that consolidating all static OS files into /usr
> > (removing the distinction between /usr and the static parts of the root
> > filesystem) is the route that Debian is taking. I think we do not have
> > consensus on how that is to be achieved.
>
> Is everybody persuaded now that there is
> no point in continuing to support non-merged systems?

No. Sorry, I phrased that badly. The consensus that I think we have is:
we are no longer attempting to support systems booting without /usr
mounted, and therefore it is not a bug if programs and libraries on the
rootfs have dependencies in /usr. (That's a less strong guarantee than
the one you are probably hoping for.)

That means that libraries and executables whose paths are not hard-coded
anywhere are allowed to migrate from the root filesystem to /usr (modulo
warnings about possible bugs: see elsewhere in this thread).

However, it doesn't give us a solution for what should happen to things
that are canonically on the root filesystem and *do* have their absolute
paths hard-coded somewhere, most critically /lib*/ld*.so.* and /bin/sh.

One of those mechanisms to consolidate files into /usr is to redirect files
en masse "behind dpkg's back", making use of dpkg's support for unpacking
through symlinks-to-directories. This is how debootstrap --merged-usr
works, and also how a system with usrmerge installed continues to work
afterwards. It has the advantage that it works for all files, even those
that get their paths hard-coded in various places: after undergoing this
transition, /PATH and /usr/PATH can be used interchangeably (for any
PATH in one of the directories that was historically on the rootfs).
This is the result that I think most people mean when they say "the /usr
merge" or "merged /usr".

The other mechanism is migrating things from the rootfs into /usr
individually, on a per-package basis. GLib's move from /lib/MULTIARCH
to /usr/lib/MULTIARCH is a typical example. I think it's misleading to
refer to this as "merged /usr" or "/usr merge" - although it can be
*a step towards* that - because in most cases, after this transition,
the two paths are *not* equivalent (one exists, the other doesn't),
unless the first mechanism has *also* been used.

> Libraries are easy to move since the linker will find them no matter 
> where they are installed, but moving the binaries too will require 
> merging older systems.

Yes.

I think some people (perhaps including Guillem?) might be advocating
including executables in the second mechanism described above by
moving them, on a per-package basis, to the root filesystem, and
creating compatibility symlinks on the root filesystem if it has not
undergone the /usr merge, like /bin/plymouth -> /usr/bin/plymouth and
/sbin/iptables -> /usr/sbin/iptables. However, so far, this is rare
(plymouth and iptables are among only a few examples on my laptop)
and if adopted, it seems likely to be a slow transition: most of the
affected packages are exactly those that are critical for boot, so their
maintainers tend to be quite conservative. One thing to watch out for
with this approach is that the symlinks must be created from the
maintainer scripts instead of being shipped in the data.tar.*, because
otherwise they would break the usrmerge-style approach to achieving
merged /usr.

Another issue with this approach is that it has quite a lot of moving parts
(see recent upgrade issues with libgcc1 and libgcc-s1 for an example of
how even a very experienced maintainer can reach unexpected situations in
a transition like this).

smcv

Bug#951616: ITP: python-libais -- Library for decoding maritime Automatic Identification System messages

[Adam Cecile]

Package: wnpp
Severity: wishlist
Owner: Adam Cecile 

* Package name: python-libais
  Version : 0.17+git.20190917.master.e464cf8
  Upstream Author : Kurt Schwehr
* URL : https://github.com/schwehr/libais
* License : Apache-2.0 / BSD-3-Clause
  Programming Lang: C++ / Python
  Description : Library for decoding maritime Automatic Identification
System messages

 There are two interfaces to libais, one high-level iterator based one
 and a low-level fast C++ only one.
 .
 To use the low-level C++ interface directly, you need to handle multi-line
 messages and padding yourself.

I intend to maintain this package within the DPMT.

Re: Best practices for Debian developers who are also upstreams?

[Antoine Beaupré]

On 2020-02-08 22:07:48, Otto Kekäläinen wrote:
> Hello!
>
> I've ended up in being both the maintainer in Debian and an upstream
> developer for a couple of packages and I have been fantasizing about
> how to optimize my workflow so that I primarily fix all bugs and do QA
> directly on the upstream development version (=upstream git master)
> and then have only a very small overhead work then importing and
> uploading new upstream releases in Debian.
>
> Is somebody else already doing something similar like this?
> Any tips, blogs, examples on the topic?

I maintain a few packages in Debian for which I am also the (mostly
solo) upstream:

 * feed2exec
 * gameclock
 * monkeysign
 * undertime

Those packages are all *native* packages, in that they *don't* have a -N
suffix in the version number. This is generally frowned upon, but in my
situation, it's been helpful because I don't need to worry about
"upstream" vs "debian": everything is "in Debian".

This implies that I have only one debian/ directory: the upstream. When
I need to make a release, I push it to unstable. If I need to make a
hotfix release for security or stable, I make a branch. I use semantic
versionning and basically maintain a "major" version per Debian release
(as necessary, which is "rarely").

> I find it annoying to carry Debian patches for bugs that could be
> fixed globally at upstream, and it is also annoying when something
> Debian QA catches is broken at upstream and I find it out only at the
> time I am preparing an upload to Debian.

I do both at once, all the time, as part of CI and the release
process. I do the Debian checklist before I even push tags upstream, but
sometimes things come up in Debian after I pushed the tag. Then I just
make another patch release, no big deal.

> My goals would be:
> - have debian/ in upstream repository, and a CI that does the bulk of
> Debian QA on every upstream commit and immediately gives feedback to
> upstream devs that "break" something from Debian QA point of view

The tricky part here is generating a new version without mangling
debian/changelog all the time. I haven't found a great story for that
that works with git, but maybe you can generate syntetic commits to fake
a new Debian package version in CI.

> - have the delta of Debian debian/ an upstream debian/ as small as
> possible

For me, there's no diff. It's the same branch, same code. I do, however,
generate debian/changelog only on release so I don't have to repeat
myself between commit logs (in git) and release notes (in
debian/changelog).

> - fix all bugs detected in Debian directly at upstream when possible

Not sure what that means, but when I fix a bug in Debian, it's fixed in
"upstream" first, ie. I push it to git, and I tag it as "pending" in the
BTS to make it clear it will be fixed in the next upload.

> - when not possible, fix them "locally" first in Debian and eventually
> have it upstreamed

I never have to do that. To fix it "first in Debian", I just make a new
upstream release. If it's on a previous "branch" (e.g. stable), I make a
release on that branch.

For example right now undertime is 1.7.0 in buster and 1.8.0 in
bullseye/sid. Next RC bugfix release is 1.8.1 in sid and 1.7.1 in
buster, and I would have a 1.7.x branch to follow buster (which I
haven't needed yet).

> - have it easy to compare Debian and upstream debian/ contents

Trivial. It's my different maintenance branches.

> - bonus: import upstream releases as git operations without having to
> export and import tar.gz packages

I export tarballs only as part of the release process and otherwise
don't bother with those artefacts at all.

> I have in rdiff-backup pretty much this setup already in place but I
> have some challenges on how to handle the debian/changelog file and a
> couple of other details. I would be very keen to learn from others
> before inventing too much of new.

The changelog is the hard part. I update it only on release which
simplifies things tremendously, but will make CI harder, as I mentioned
before.

Also, this is not for everyone: it makes it hard for Debian folks to
send updates to the git repository (because it might not be on
debian.org infra) and inversely it makes it hard for "upstream" to make
a release if they're not Debian developers. Fortunately, I am both and
those are mostly solo projects so that hasn't been an issue so far.

When it will be, the package will just go non-native and follow more
usual development practices. This happened to etckeeper when joeyh left
us, for example.

I hope that helps!

A.

-- 
Si les triangles avaient un Dieu, ils lui donneraient trois côtés.
- Montesquieu, Lettres persanes

Re: Is there still a point in installing libgcrypt to /lib instead of /usr/lib

[Marco d'Itri]

On Feb 18, Simon McVittie  wrote:

> No. Sorry, I phrased that badly. The consensus that I think we have is:
> we are no longer attempting to support systems booting without /usr
> mounted, and therefore it is not a bug if programs and libraries on the
> rootfs have dependencies in /usr. (That's a less strong guarantee than
> the one you are probably hoping for.)
We do not just have a consensus, this has also been a fact for a long 
time now:

kmod (25-2) unstable; urgency=medium

  * Moved the libraries to /usr/lib/. (Closes: #894566)

 -- Marco d'Itri   Sat, 17 Nov 2018 01:56:00 +0100

> However, it doesn't give us a solution for what should happen to things
> that are canonically on the root filesystem and *do* have their absolute
> paths hard-coded somewhere, most critically /lib*/ld*.so.* and /bin/sh.
This does not matter as long as we have to support un-merged-/usr 
systems.

> I think some people (perhaps including Guillem?) might be advocating
> including executables in the second mechanism described above by
> moving them, on a per-package basis, to the root filesystem, and
> creating compatibility symlinks on the root filesystem if it has not
> undergone the /usr merge, like /bin/plymouth -> /usr/bin/plymouth and
> /sbin/iptables -> /usr/sbin/iptables. However, so far, this is rare
> (plymouth and iptables are among only a few examples on my laptop)
> and if adopted, it seems likely to be a slow transition: most of the
Slow, and also a lot of work for no significant benefits.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: How to undo a merged-user installation?

[Sam Hartman]

Hi.
I don't know of a way to do a install from the menus in d-i, but you can
use the --no-merged-usr switch to debootstrap.
Presumably you could boot d-i or at least a live system and run
debootstrap manually from a shell.

Re: Is there still a point in installing libgcrypt to /lib instead of /usr/lib

[Andreas Henriksson]

Hello,

On Tue, Feb 18, 2020 at 10:10:05PM +0100, Marco d'Itri wrote:
> On Feb 18, Simon McVittie  wrote:
> 
> > No. Sorry, I phrased that badly. The consensus that I think we have is:
> > we are no longer attempting to support systems booting without /usr
> > mounted, and therefore it is not a bug if programs and libraries on the
> > rootfs have dependencies in /usr. (That's a less strong guarantee than
> > the one you are probably hoping for.)
> We do not just have a consensus, this has also been a fact for a long 
> time now:

.. and is completely unrelated to merged-usr (?!).

> 
> kmod (25-2) unstable; urgency=medium
> 
>   * Moved the libraries to /usr/lib/. (Closes: #894566)
> 
>  -- Marco d'Itri   Sat, 17 Nov 2018 01:56:00 +0100
> 
> > However, it doesn't give us a solution for what should happen to things
> > that are canonically on the root filesystem and *do* have their absolute
> > paths hard-coded somewhere, most critically /lib*/ld*.so.* and /bin/sh.
> This does not matter as long as we have to support un-merged-/usr 
> systems.
> 
> > I think some people (perhaps including Guillem?) might be advocating
> > including executables in the second mechanism described above by
> > moving them, on a per-package basis, to the root filesystem, and
> > creating compatibility symlinks on the root filesystem if it has not
> > undergone the /usr merge, like /bin/plymouth -> /usr/bin/plymouth and
> > /sbin/iptables -> /usr/sbin/iptables. However, so far, this is rare
> > (plymouth and iptables are among only a few examples on my laptop)
> > and if adopted, it seems likely to be a slow transition: most of the
> Slow, and also a lot of work for no significant benefits.

As I see it there are two possible ways forward:

a/ make merged-usr mandatory in one stable release and in the next
   we can just install the files under /usr. No per-package symlinks
   needed.
b/ someone volunteers to write a debhelper addon which runs after
   dh_install, detects files in /lib, /bin and /sbin, moves them
   into /usr and generates the needed postinst code doing the compat
   symlinks for non-merged systems.


The third option that I see as a no-go is that each package maintainer
implements this on their own because:
1. Getting this finished will take way too long.
2. Manually written maintainer scripts is well known source of bugs.
3. Broken maintainer scripts are a horrible user experience.
4. People will keep repeating the mistake of shipping the symlinks
   in the package (once they realize manually written maintainer scripts
   are horrible?!) and thus break their package on merged-usr systems.
5. Doing the work this way will consume too much resources for no gain.
6. People being fine with the current status quo will not like 5/
   further contributing to 1/.


I think it's time we either try to figure out if/how we can actually
get a decision on doing a/ soon, or for those who can't live with
one release making merged-usr mandatory then start working on
that debhelper needed to do b/ now!

(Also when/if doing b/ or the 'no-go' option someone might want to think
about how per-package symlinks generated in postinst is going to play
nice with Essential: yes packages which are supposed to work
unconfigured.)

Regards,
Andreas Henriksson

Re: Best practices for Debian developers who are also upstreams?

[Simon McVittie]

On Tue, 18 Feb 2020 at 15:47:17 -0500, Antoine Beaupré wrote:
> The tricky part here is generating a new version without mangling
> debian/changelog all the time. I haven't found a great story for that
> that works with git, but maybe you can generate syntetic commits to fake
> a new Debian package version in CI.

I think the way to do this is to have an uncommitted change to d/changelog
when you do your build, and choose your CI version numbers carefully so
that they interleave correctly with "real" version numbers.
https://gitlab.collabora.com/smcv/deb-build-snapshot/ (currently on my
employer's infrastructure, but I'll probably move it to Salsa at some
point) is designed for this workflow.

The version-number-generating part, deb-git-version-gen, implements the
careful choice of version numbers and has been split out into a separate
script which I might propose for inclusion in devscripts at some point -
it should be feasible to run as a git-buildpackage hook or similar,
and can either run dch itself, or output a simple version number or a
blob of JSON describing the package's versioning (the latter includes
some suggested arguments for dch).

> Also, this is not for everyone: it makes it hard for Debian folks to
> send updates to the git repository (because it might not be on
> debian.org infra)

This seems particularly weird for native packages, which are supposedly
"written specifically to be a Debian package" (Policy §5.6.12). When
maintaining upstream packages that include their own Debian-compatible
packaging, I sometimes make them non-native with versions like
1.2.3-0upstream1, which makes it trivial to repackage them as non-native
elsewhere.

> and inversely it makes it hard for "upstream" to make
> a release if they're not Debian developers.

This was a practical problem when Joey Hess left Debian: ikiwiki was a
native package, with no release procedure other than uploading it to
Debian, and I became a single point of failure for security releases
because I was suddenly the only person with both ikiwiki commit access
and Debian upload rights.

(I later disentangled it so that upstream releases are just git tags
pointing to a native package, and the Debian packaging is non-native.)

smcv

tmpfiles.d and docker images (was Re: opentmpfiles & opensysusers, and its use in the Debian policy)

[Michael Hudson-Doyle]

On Fri, 3 Jan 2020 at 06:29, Simon McVittie  wrote:

> On Fri, 03 Jan 2020 at 00:05:10 +0800, Shengjing Zhu wrote:
> > The advantages of sysusers.d and tmpfiles.d are that you don't need to
> > call some magic scripts.
> > You only need to write declarative configuration files.
>
> Individual packages only need to install declarative configuration files,
> but the OS distribution infrastructure needs to do something to make
> the contents of those configuration files take effect (and, crucially,
> know when the implementation has finished running).
>
> On systemd systems, that's approximately:
>
> - run systemd-tmpfiles when a package installs a tmpfiles.d snippet
>   (this is added to the package's postinst by dh_installsystemd)
>
> - run systemd-sysusers when a package installs a sysusers.d snippet
>   (I don't think we have tools to add this to the postinst yet, because
>   packages are currently meant to run adduser --system instead, but
>   more-systemd-centric distributions probably already do this in their
>   equivalent of the postinst)
>
> - run systemd-tmpfiles during boot (this is systemd-tmpfiles-setup.service,
>   part of systemd)
>
> - run systemd-sysusers during boot (this is
>   systemd-sysusers.service, part of systemd)
>
> The opentmpfiles and opensysusers packaging will need to arrange to do
> something analogous, most likely in cooperation with dh_installsystemd
> or some other debhelper step for the first two points, and with LSB init
> scripts for the tasks where systemd uses one-shot services.


So in Ubuntu we got this interesting bug
https://bugs.launchpad.net/ubuntu/+source/haproxy/+bug/1855140 which can be
summarized as saying that haproxy doesn't work out of the box in a docker
container, because it installs a tmpfiles.d snippet but nothing processes
it (I haven't tried, but I very much expect that the behaviour is the same
between Debian and Ubuntu here).

On the face of it, this is a bug in the package: it depends for its
operation on a package not listed in Depends but making the package depend
on systemd wouldn't actually be very useful for getting it running in
docker, for a few reasons. I think it would be better to change debhelper:

 * to add "systemd | opentmpfiles" to misc:Depends whenever a package
installs a tmpfiles.d snippet
 * stop checking for -d /run/systemd/system in
autoscripts/postinst-init-tmpfiles and check for systemd-tmpfiles /
opentmpfiles in $PATH instead (and using the one that is found, obviously)

Obviously there are a few variations on the above, like:

 * splitting a systemd-tmpfiles package out of the systemd package first
 * change systemd(-tmpfiles) and opentmpfiles to Provides: tmpfiles and use
alternatives to provide /usr/bin/tmpfiles, have debhelper reference these
instead

But the approach I outlined seems simplest and easiest to implement. Does
it make sense to people here? I can try to work on a patch (although my
perl isn't the greatest).

Cheers,
mwh

Re: Is there still a point in installing libgcrypt to /lib instead of /usr/lib

[Guillem Jover]

On Sun, 2020-02-16 at 11:59:56 +, Simon McVittie wrote:
> I would be grateful if people who advocate transitioning individual
> packages, and people who consider the approach taken by usrmerge and
> debootstrap to be sufficient, could refer to their preferred route in a
> way that makes it clear which one they are advocating. Saying we should
> do a transition "properly" is tautologous - of course we should! - but
> when people disagree about what the proper way to do it is, it becomes an
> ambiguous recommendation that doesn't guide anyone to do the right thing.

I've been consistently calling the concept of merging /* into /usr/* as
merged-/usr and the specific approach of using directory symlinks as
merged-/usr-via-symlinks (although I think that's confusing as the
other approach does use symlink farms), so I think using either
merged-/usr-via-aliased-dirs or merged-/usr-via-symlink-dirs is more
clear (will be renaming the buildinfo tainted tag). The approach I've
been proposing I'd call merged-/usr-via-moves-and-symlink-farms or
something along those lines.

I've dumped this all into 
and updated the Dpkg FAQ:



> The approach that transitions individual packages solves some bugs
> (notably, if you ask dpkg which package owns /usr/lib/MA/libfoo.so.0,
> you get the right answer, which has a lot of desirable consequences). It
> also appears to *cause* some bugs. The ones I know about are:
> 
> * packages that hard-code paths into /lib stop working on systems that
>   have *not* undergone the usrmerge-style /usr merge, because
>   /lib/MA/libfoo.so.0 no longer exists (for example see #950715
>   involving libgcc.so.1 and the gold linker);

For any pathname that has been hardcoded a symlink can be used for
backwards compat, nothing unlike /bin or /sbin here. This looks just
like a normal bug from a botched transition, nothing special.

> * there is a class of bugs on systems that have *not* undergone the
>   usrmerge-style /usr merge, involving old libraries lingering in /lib/MA
>   (see #949395 for a summary of the instances that I know about), which
>   are very hard to debug because they are unreproducible, to do with the
>   state of an individual system, and are related to upgrades that happened
>   years in the past and whose logs expired long ago;

merged-/usr (in general) seems irrelevant here, if this is really a bug
in dpkg, then it needs fixing regardless of any approach taken, as there
is nothing in dpkg really distinguishing between / and /usr as it is
pathname neutral.

I've pending to comment on that bug, but I was skimming over the involved
code and didn't see anything obviously wrong that would forget pathnames.
But my first suspect would be that we are not doing fsync() on the parent
directories, the second would be perhaps related to multiarch refcounting
logic, but ISTR this having happened in old glibc perhaps before multiarch
was deployed, but would need to check.

> * when paths migrate between package names and between paths at the same
>   time, there can be undetected file conflicts on systems that *have*
>   undergone the usrmerge-style /usr merge (for example see #950624,
>   again in libgcc.so.1)

This is one of the issues inherent in merged-/usr-via-aliased-dirs.

> In Debian we have (as usual) made life more difficult for ourselves by the
> /usr merge being optional,

I fully agree, but not on the reason for the root cause. Allowing the
merged-/usr-via-aliased-dirs hack at all, has meant that we cannot do
fully automatic migrations via debhelper with no maintainer scripts
involved whatsoever. :(

> which means that in any transition or upgrade
> scenario, maintainers have to consider two cases: one where the system
> has undergone the /usr merge, and one where it has not. For example,
> #950624 only happens on systems that have undergone the /usr merge,
> while #950715 and #949395 only happen on systems that have not. Testing
> on any single system cannot detect all of these: to detect all the bugs,
> we have to try both configurations.

The difference between these issues IMO is one between bugs (a botched
migration with no compat symlinks, and a potential dpkg bug), and design
flaws inherent in the merged-/usr-via-aliased-dirs approach.

I'm not sure how anyone could claim that the merged-/usr-via-aliased-dirs
approach is not a hack, even if it might have some possible benefits,
when it circumvents the package manager completely (we might as well
ditch it and go use bare tarballs or something :/…), and implies action
at a distance where the filesystem layout is injected from something
that is not packaged at all and should not be packaged (or we'd be forced
to hardcode bootstrapping ordering into each bootstrapper forever) so we
also lose locality from the .deb.

That approach would be acceptable in a distant future once and iff w

Re: Is there still a point in installing libgcrypt to /lib instead of /usr/lib

[The Wanderer]

On 2020-02-18 at 20:50, Guillem Jover wrote:

> On Sun, 2020-02-16 at 11:59:56 +, Simon McVittie wrote:
> 
>> I would be grateful if people who advocate transitioning
>> individual packages, and people who consider the approach taken by
>> usrmerge and debootstrap to be sufficient, could refer to their
>> preferred route in a way that makes it clear which one they are
>> advocating. Saying we should do a transition "properly" is
>> tautologous - of course we should! - but when people disagree about
>> what the proper way to do it is, it becomes an ambiguous
>> recommendation that doesn't guide anyone to do the right thing.
> 
> I've been consistently calling the concept of merging /* into /usr/*
> as merged-/usr and the specific approach of using directory symlinks
> as merged-/usr-via-symlinks (although I think that's confusing as
> the other approach does use symlink farms), so I think using either
> merged-/usr-via-aliased-dirs or merged-/usr-via-symlink-dirs is more
> clear (will be renaming the buildinfo tainted tag). The approach
> I've been proposing I'd call merged-/usr-via-moves-and-symlink-farms
> or something along those lines.

As a tangent, because this has never made sense to me:

Is there a reason this is all looking to merge /* into /usr/* instead of
the other way around?

The latter looks to me as if it would ultimately make it possible to
drop the /usr directory entirely (once we've had a release or three in
which it contains nothing but symlinks) - whereas the former means we'd
have the extra four characters at the head of nearly every path to
installed software forever, for no apparent benefit, and would also seem
to mean we'd have to keep the symlinks in / around potentially forever,
even after the transition has been complete for yonks.

I'm not sure I'd support dropping /usr, but I don't think I can remember
having ever run across (or come up with) any arguments against it which
don't seem like they would also be arguments against doing this merge at
all, so the fact that the merge is apparently being done in this
direction has consistently puzzled me.

Does keeping /usr have some kind of benefit in its own right which I'm
not seeing, even for cases when it's on the root filesystem?

(Answers in the forms of pointers to Websites, or even to archives of
past discussion where this would be made clear, are entirely
acceptable.)

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw



signature.asc
Description: OpenPGP digital signature

Re: Is there still a point in installing libgcrypt to /lib instead of /usr/lib

[Boyuan Yang]

在 2020-02-18二的 23:58 -0500，The Wanderer写道：
> On 2020-02-18 at 20:50, Guillem Jover wrote:
> 
> > On Sun, 2020-02-16 at 11:59:56 +, Simon McVittie wrote:
> > 
> > > I would be grateful if people who advocate transitioning
> > > individual packages, and people who consider the approach taken by
> > > usrmerge and debootstrap to be sufficient, could refer to their
> > > preferred route in a way that makes it clear which one they are
> > > advocating. Saying we should do a transition "properly" is
> > > tautologous - of course we should! - but when people disagree about
> > > what the proper way to do it is, it becomes an ambiguous
> > > recommendation that doesn't guide anyone to do the right thing.
> > 
> > I've been consistently calling the concept of merging /* into /usr/*
> > as merged-/usr and the specific approach of using directory symlinks
> > as merged-/usr-via-symlinks (although I think that's confusing as
> > the other approach does use symlink farms), so I think using either
> > merged-/usr-via-aliased-dirs or merged-/usr-via-symlink-dirs is more
> > clear (will be renaming the buildinfo tainted tag). The approach
> > I've been proposing I'd call merged-/usr-via-moves-and-symlink-farms
> > or something along those lines.
> 
> As a tangent, because this has never made sense to me:
> 
> Is there a reason this is all looking to merge /* into /usr/* instead of
> the other way around?

Sure. Before we repeat all discussions that we previously made again, I invite
everyone currently reading this thread to take a look at Debian's Technical
Committee's last decision again:

* https://bugs.debian.org/914897
* https://lists.debian.org/debian-devel-announce/2019/03/msg1.html

(Apologize that this TC decision was not reflected on www.debian.org; it is a
long overdue for the debian-www team... Merge Requests welcome!)

I hope this could provide more background to you and further discussions can
be made on top of the TC decision.

-- 
Thanks,
Boyuan Yang


signature.asc
Description: This is a digitally signed message part

Re: Is there still a point in installing libgcrypt to /lib instead of /usr/lib

[Marco d'Itri]

On Feb 19, The Wanderer  wrote:

> Is there a reason this is all looking to merge /* into /usr/* instead of
> the other way around?
Yes, nicely documented by the links collected here: 
https://wiki.debian.org/UsrMerge .

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Is there still a point in installing libgcrypt to /lib instead of /usr/lib

[Marco d'Itri]

On Feb 19, Guillem Jover  wrote:

> For any pathname that has been hardcoded a symlink can be used for
> backwards compat, nothing unlike /bin or /sbin here. This looks just
> like a normal bug from a botched transition, nothing special.
Creating symlinks in /bin and /sbin DOES NOT result in a merged-/usr 
system, because the content of /usr would not be decoupled anymore from 
the content of /.
A merged-/usr system must have /bin /sbin /lib* symlinks to /usr.

What you are proposing is NOT an alternative implementation of 
merged-/usr but something else, which has no significant benefits.

-- 
ciao,
Marco


signature.asc
Description: PGP signature