Re: Build libzstd using cmake; add a non-cmake build profile?

2022-12-26 Thread Peter Pentchev 
On Mon, Dec 26, 2022 at 08:37:51AM +0200, Peter Pentchev wrote:
> Hi,
> 
> In #1020403 there is a request to install the CMake build glue for
> the zstd library in its -dev package. I think that this is a good
> idea, and I have a pretty much ready-for-uploading set of changes
> that do that. For the purposes of this discussion I have uploaded
> the proposed "switch to CMake and install the CMake build glue"
> commits to my own fork of the libzstd repository at
>   https://salsa.debian.org/roam/libzstd
> 
> The main thing that gave me a bit of a pause before running
> `dgit push-source` (and *thanks* for making it that easy!) is that
> right now libzstd is effectively an essential package (ObXThread:
> not in the Policy sense) since dpkg pre-depends on it to support
> e.g. recent Ubuntu debs that use zstd as the compression method for
> the data part of the package. So... if I introduce a build-time
> dependency on CMake, this will probably create a non-trivial dependency
> loop for people who try to bootstrap Debian onto new architectures.
> Does this mean that it would be best for me to include an e.g. stage1
> build profile that does not build-depend on cmake and falls back to
> the old/current way of building directly using `make`?
> 
> Hm, but if I do that, it seems that it might be best to put the CMake
> build glue files into a separate package and make libzstd-dev depend on
> that new package in the  case, since IIUC the stage1 build
> profile is not allowed to change the contents of a binary package, so
> I cannot simply drop all the files in the /usr/lib//cmake/ directory.
> That's not a big hurdle; if people say that this is the way to go, then
> this is what will be done.
> 
> Many thanks to all the people who keep working on all the tools and
> toolchains that make this message make sense at all!

Hm, I just thought of a hybrid solution, but I don't think it would be
a really, really good idea: keep the non-cmake build, but also invoke
cmake with a modified lists file so that it *only* generates the build
glue; then still put that in a separate package, all of that governed
by a !stage1 build profile. I'm not sure I like it a lot, since that
would mean that the shipped library is not actually compiled using CMake,
so one might say the cmake build glue is a lie... but it could be
an option if people think it's best.

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.net r...@debian.org p...@storpool.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Re: Build libzstd using cmake; add a non-cmake build profile?

2022-12-26 Thread Timo Röhling 

* Peter Pentchev  [2022-12-26 08:37]:

Hi,

In #1020403 there is a request to install the CMake build glue for
the zstd library in its -dev package. I think that this is a good
idea, and I have a pretty much ready-for-uploading set of changes
that do that. For the purposes of this discussion I have uploaded
the proposed "switch to CMake and install the CMake build glue"
commits to my own fork of the libzstd repository at
 https://salsa.debian.org/roam/libzstd

The main thing that gave me a bit of a pause before running
`dgit push-source` (and *thanks* for making it that easy!) is that
right now libzstd is effectively an essential package (ObXThread:
not in the Policy sense) since dpkg pre-depends on it to support
e.g. recent Ubuntu debs that use zstd as the compression method for
the data part of the package. So... if I introduce a build-time
dependency on CMake, this will probably create a non-trivial dependency
loop for people who try to bootstrap Debian onto new architectures.
Does this mean that it would be best for me to include an e.g. stage1
build profile that does not build-depend on cmake and falls back to
the old/current way of building directly using `make`?

Hm, but if I do that, it seems that it might be best to put the CMake
build glue files into a separate package and make libzstd-dev depend on
that new package in the  case, since IIUC the stage1 build
profile is not allowed to change the contents of a binary package, so
I cannot simply drop all the files in the /usr/lib//cmake/ directory.
That's not a big hurdle; if people say that this is the way to go, then
this is what will be done.

Many thanks to all the people who keep working on all the tools and
toolchains that make this message make sense at all!


I appreciate your concern for the implications for the Debian
ecosystem as a whole; in this particular case, I believe I can put
your mind at ease: cmake has a bootstrap profile already, because
dependency loops have come up before (e.g. libjsoncpp).

I am Cc'ing debian-cross in case I overlooked something important,
but I am reasonably sure that you need not deal with this issue in
libzstd.


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Build libzstd using cmake; add a non-cmake build profile?

2022-12-26 Thread Andrea Pappacoda 
Il giorno lun 26 dic 2022 alle 08:37:51 +02:00:00, Peter Pentchev 
 ha scritto:

In #1020403 there is a request to install the CMake build glue for
the zstd library in its -dev package. I think that this is a good
idea, and I have a pretty much ready-for-uploading set of changes
that do that. For the purposes of this discussion I have uploaded
the proposed "switch to CMake and install the CMake build glue"
commits to my own fork of the libzstd repository


Hi Peter. Apart from your bootstrapping concerns, please keep in mind 
that upstream's CMake build script is not official; citing the README:


> make is the officially maintained build system of this project. All 
other build systems are "compatible" and 3rd-party maintained, they may 
feature small differences in advanced options. When your system allows 
it, prefer using make to build zstd and libzstd.


Using CMake instead of make has caused some interesting issues in the 
past. The first that comes to my mind happened in Arch Linux, and got 
featured on the Phoronix news site: 



Since the make build generates a pkg-config file, you could look into 
leveraging that instead. You could either send a patch to the CMake 
projects using libzstd adding a Find module which calls 
pkg_search_module(), as mentioned in #1020403, or patch zstd's 
makefiles to install a small Find module itself (this is not really a 
good practice, as ideally upstreams should install CMake Config files, 
but should work nonetheless). Alternatively, you could even submit a 
patch to CMake adding The Find module there; CMake already ships a lot 
of them.

Re: Build libzstd using cmake; add a non-cmake build profile?

2022-12-26 Thread Peter Pentchev 
On Mon, Dec 26, 2022 at 02:00:31PM +0100, Andrea Pappacoda wrote:
> Il giorno lun 26 dic 2022 alle 08:37:51 +02:00:00, Peter Pentchev
>  ha scritto:
> > In #1020403 there is a request to install the CMake build glue for
> > the zstd library in its -dev package. I think that this is a good
> > idea, and I have a pretty much ready-for-uploading set of changes
> > that do that. For the purposes of this discussion I have uploaded
> > the proposed "switch to CMake and install the CMake build glue"
> > commits to my own fork of the libzstd repository
> 
> Hi Peter. Apart from your bootstrapping concerns, please keep in mind that
> upstream's CMake build script is not official; citing the README:
> 
> > make is the officially maintained build system of this project. All other
> build systems are "compatible" and 3rd-party maintained, they may feature
> small differences in advanced options. When your system allows it, prefer
> using make to build zstd and libzstd.

Yes, I am aware of this, and I do take additional care to make sure that
things work as expected. Still, thanks for mentioning this, and see below...

> Using CMake instead of make has caused some interesting issues in the past.
> The first that comes to my mind happened in Arch Linux, and got featured on
> the Phoronix news site:
> 

Right... thanks for reminding me of that! I had indeed noticed a couple of days
ago that the CMake build differed in a couple of minor ways from the Makefile 
one
and I had made a mental note to fix that... and then I forgot. There were two
main differences: legacy support and setting the C standard. The former is now
handled via a CMake config option passed via `dh_auto_configure`, and the 
latter is
disabled with a patch taken from the upstream development branch, although in
my (limited) testing there was no effect on the compression speed, at least with
the GCC version in Debian unstable as of now.

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.net r...@debian.org p...@storpool.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Re: Build libzstd using cmake; add a non-cmake build profile?

2022-12-26 Thread Simon McVittie 
On Mon, 26 Dec 2022 at 14:00:31 +0100, Andrea Pappacoda wrote:
> or patch zstd's makefiles to install a small Find
> module itself (this is not really a good practice, as ideally upstreams
> should install CMake Config files, but should work nonetheless).

If you're considering patching zstd's makefiles, I believe the preferred
route is to install CMake "config modules" containing an imported target,
like dbus and libsdl2 do. The one in dbus is a reasonably simple example
of wrapping a pkg-config module with a CMake config module, while the one
in libsdl2 bypasses pkg-config and provides the equivalent information
more directly. Either way, this is something that would make sense to
contribute upstream.

Both dbus and libsdl2 have optional-but-not-recommended CMake build
systems, similar to zstd, which we don't use in Debian; we follow upstream
recommendations to build dbus with Autotools (in older branches) or Meson
(in the 1.15.x development branch), and libsdl2 with Autotools. The
Autotools and Meson build systems for those projects also install a
simplified CMake config module, which can be consumed by CMake projects
in the same way as if dbus/libsdl2 had themselves been built with CMake.
The CMake config module is generated from a template using @variable@
substitutions.

In older versions of libsdl2 the config module is not completely compatible
with what its CMake build system would have installed, but the one in
bookworm should be reasonably feature-complete.

If dependent projects are expected to call find_package(Foo), then the
CMake config module should be installed as either
${libdir}/cmake/Foo/FooConfig{,Version}.cmake in mixed-case (like dbus
does) or ${libdir}/cmake/Foo/foo-config{,-version}.cmake in lower-case
(like libsdl2 does). I don't know whether one of those is preferred over
the other.

It is a good idea to have a test for each supported use-case in the
package's autopkgtests. dbus and libsdl2 have some tests for this which
might make useful inspiration.

smcv

Help trying to debug an sbuild failure?

2022-12-26 Thread Theodore Ts'o 
Hi, I'm trying to figure out an sbuild failure on my laptop.  The
sbuild environment from replicated from my desktop where things work
perfectly well.  But in my laptop, things are failing at the 
"Setup apt archive" step with

   E: Failed to change to directory ‘/<>’: Permission denied

And I'm completely at a loss trying to figure out what might be going
wrong.  Can anyone give me some hints about what to look for?

Thanks,

- Ted



sbuild (Debian sbuild) 0.84.2 (08 December 2022) on letrec.thunk.org

+==+
| f2fs-tools 1.15.0-1 (amd64)  Mon, 26 Dec 2022 19:20:17 + |
+==+

Package: f2fs-tools
Version: 1.15.0-1
Source Version: 1.15.0-1
Distribution: unstable
Machine Architecture: amd64
Host Architecture: amd64
Build Architecture: amd64
Build Type: full

I: NOTICE: Log filtering will replace 
'var/run/schroot/mount/unstable-amd64-sbuild-a67a3165-6688-4368-a376-66e094e41dfa'
 with '<>'
I: NOTICE: Log filtering will replace 'build/f2fs-tools-cVTRAh/resolver-CwG6Va' 
with '<>'

+--+
| Update chroot|
+--+

Get:1 http://httpredir.debian.org/debian unstable InRelease [161 kB]
Get:2 http://httpredir.debian.org/debian unstable/main Sources.diff/Index [63.6 
kB]
Get:3 http://httpredir.debian.org/debian unstable/main amd64 
Packages.diff/Index [63.6 kB]
Get:4 http://httpredir.debian.org/debian unstable/main Sources 
T-2022-12-26-1404.34-F-2022-12-26-0804.45.pdiff [15.0 kB]
Get:4 http://httpredir.debian.org/debian unstable/main Sources 
T-2022-12-26-1404.34-F-2022-12-26-0804.45.pdiff [15.0 kB]
Get:5 http://httpredir.debian.org/debian unstable/main amd64 Packages 
T-2022-12-26-1404.34-F-2022-12-26-0804.45.pdiff [33.2 kB]
Get:5 http://httpredir.debian.org/debian unstable/main amd64 Packages 
T-2022-12-26-1404.34-F-2022-12-26-0804.45.pdiff [33.2 kB]
Fetched 337 kB in 1s (238 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

+--+
| Fetch source files   |
+--+


Local sources
-

/tmp/gbp/f2fs-tools_1.15.0-1.dsc exists in /tmp/gbp; copying to chroot
I: NOTICE: Log filtering will replace 
'build/f2fs-tools-cVTRAh/f2fs-tools-1.15.0' with '<>'
I: NOTICE: Log filtering will replace 'build/f2fs-tools-cVTRAh' with 
'<>'

+--+
| Install package build dependencies   |
+--+


Setup apt archive
-

Merged Build-Depends: debhelper-compat (= 13), libblkid-dev, libselinux1-dev, 
pkg-config, uuid-dev, build-essential, fakeroot
Filtered Build-Depends: debhelper-compat (= 13), libblkid-dev, libselinux1-dev, 
pkg-config, uuid-dev, build-essential, fakeroot
E: Failed to change to directory ‘/<>’: Permission denied
I: The directory does not exist inside the chroot.  Use the --directory option 
to run the command in a different directory.
Dummy package creation failed
E: Setting up apt archive failed

Setup apt archive
-

Merged Build-Depends: dose-distcheck
Filtered Build-Depends: dose-distcheck
E: Failed to change to directory ‘/<>’: Permission denied
I: The directory does not exist inside the chroot.  Use the --directory option 
to run the command in a different directory.
Dummy package creation failed
E: Setting up apt archive failed
E: Failed to explain bd-uninstallable

+--+
| Summary  |
+--+

Build Architecture: amd64
Build Type: full
Build-Space: n/a
Build-Time: 0
Distribution: unstable
Fail-Stage: explain-bd-uninstallable
Host Architecture: amd64
Install-Time: 0
Job: /tmp/gbp/f2fs-tools_1.15.0-1.dsc
Machine Architecture: amd64
Package: f2fs-tools
Package-Time: 0
Source-Version: 1.15.0-1
Space: n/a
Status: given-back
Version: 1.15.0-1

Finished at 2022-12-26T19:20:17Z
Build needed 00:00:00, no disk space

Re: Help trying to debug an sbuild failure?

2022-12-26 Thread Santiago Vila 

El 26/12/22 a las 20:29, Theodore Ts'o escribió:

I: The directory does not exist inside the chroot.


This is really a problem with schroot. I guess that this will not work either:

schroot -c the-chroot-name

This usually works when you are in your $HOME because this file:

/etc/schroot/default/fstab

has a line like this:

/home   /home   nonerw,bind 0   0

which makes a bind mount and makes /home to "exist" inside the chroot.

So the solution is to either try sbuild from a directory which is
bind-mounted like that, like $HOME, or add a new entry to fstab
with the directory you need.

Hope this helps.

Re: Build libzstd using cmake; add a non-cmake build profile?

2022-12-26 Thread Helmut Grohne 
Hi Timo,

On Mon, Dec 26, 2022 at 10:08:18AM +0100, Timo Röhling wrote:
> I appreciate your concern for the implications for the Debian
> ecosystem as a whole; in this particular case, I believe I can put
> your mind at ease: cmake has a bootstrap profile already, because
> dependency loops have come up before (e.g. libjsoncpp).
> 
> I am Cc'ing debian-cross in case I overlooked something important,
> but I am reasonably sure that you need not deal with this issue in
> libzstd.

I think you can drop debian-cross@l.d.o again.

There are two major ways of bootstrapping. It's the cross bootstrap
(which is what I work on) and the native bootstrap (mostly Daniel
Schepler). On debian-cross we mostly care about cross bootstrap
(surprise!). Since cmake is marked Multi-Arch: foreign, we can (almost)
always B-D on it without negatively impacting cross builds.

What probably is impacted by this dependency is native bootstrap. My
understanding of native bootstrap is too limited to explain how (severe)
it is impacted. Since apt Build-Depends: cmake, it already is entangled
in some way. I'd hope that adding another cmake dependency is not ending
the world.

If you want to add a build profile, do not use stage1. This profile has
become meaningless and is thus deprecated. We are moving to functional
build profiles. Use pkg.libzstd.nocmake or something even more
descriptive.

Last but not least, I second the idea of improving cmake such that it
can deal with pkg-config files.

Helmut

Re: Help trying to debug an sbuild failure?

2022-12-26 Thread Theodore Ts'o 
On Mon, Dec 26, 2022 at 08:45:53PM +0100, Santiago Vila wrote:
> El 26/12/22 a las 20:29, Theodore Ts'o escribió:
> > I: The directory does not exist inside the chroot.
> 
> This is really a problem with schroot. I guess that this will not work either:
> 
> schroot -c the-chroot-name
> 
> This usually works when you are in your $HOME because this file:
> 
> /etc/schroot/default/fstab

Nope, that's not the issue; yes, /tmp and /home are missing form
/etc/schroot/sbuild/fstab, but that was true on the *working* setup as
well, and from what I can tell; that's deliberate.  It looks like the
goal is to keep things hermetic when building with sbuild, so it's a
feature that there aren't random directories leaking through from the
host to the sbuild enviroment.

I think I've figured out the issue.  The problem is that the user and
group id's for sbuild are different on my desktop and my laptop, and I
did an rsync to copy the /chroot directories from my desktop to my
laptop --- and it appears that sbuild is very sensisitve about the
user id's being the same across the host and chroot environments.

Apparently sbuild copies the files for the package it is building over
a directory in /var/lib/sbuild/build, with the permissions being mode
770, and that is what sbuild bind mounts into the chroot.  If my
theory is correct, if the user/group id's are different from between
the base /etc/passwd and chroot, then things go bad in a hurry.

>From my working system (while gbp buildpackage was running sbuild):

% ls -al /var/lib/sbuild/build/
total 12
4 drwxrws--- 3 sbuild sbuild 4096 Dec 26 23:05 ./
4 drwxrws--- 4 sbuild sbuild 4096 Dec 19  2020 ../
4 drwxrwx--- 3 tytso  sbuild 4096 Dec 26 23:05 f2fs-tools-oT4KHz/

Amd on the broken (laptop) system:

# ls -al /var/lib/sbuild/build/
total 32
4 drwxrws--- 8 fwupd-refresh Debian-exim 4096 Dec 26 22:48 ./
4 drwxrws--- 4 sbuildsbuild  4096 Dec 25 14:45 ../
4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 14:01 f2fs-tools-9QfprK/
4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 16:01 f2fs-tools-btkVPv/
4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 14:20 f2fs-tools-cVTRAh/
4 drwxrwx--- 2 tytso Debian-exim 4096 Dec 26 22:48 f2fs-tools-Myld8j/
...

Each of were created from a failed sbuild invocation...  And
"Debian-exim" on my laptop has the same group id as "sbuild" on my
desktop (and which is the group id in my chroots).

This also explains the error message:

E: Failed to change to directory ‘/<>’: Permission denied

Oops.

So I guess I need to either manually juggle group id's in the chroots
(and/or my laptop's root directory --- but it's probably easier to do
it in the chroots, since there are fewer filees to chgrp in the
chroots), or I could recreate the sbuild chroots from scratch using
sbuild-createchroot (but then I would need to recreate all of the
custom hacks so that ccache,eatmydata, apt-auto-proxy, etc. are working
correctly in the chroot).

What fun

- Ted

p.s.  I guess if I had been planning ahead I would have made sure that
various system users and groups which are auto-created by packages at
install-time (and which are therefore different depending on install
order) were pre-created on my laptop with the same numerical id's as
on my desktop, since that would have avoided all *sorts* of random
problems, especially if I was going to play games with copying chroots
around --- or trying to use NFS --- and not getting taken by surprise
by this sort of thing.  Live and learn