[bug#73769] [PATCH] automake: distribute local gnulib directory modules

[Michael Pratt via Patches for Automake]

Hi all, thanks for the detailed response,

I had been discussing this with Pádraig, to at least do this manually,
and he shared with me an older letter discussing the packaging
of tools that use gnulib and how that affects, for example, debian maintainers.
I believe many of you have read this already.

https://lists.gnu.org/archive/html/bug-gnulib/2024-05/msg00124.html

Even though the concept of local modules is not explicitly mentioned,
it's very relevant to the main idea, which as I understand it, is that
even with a release tarball one should be capable of regenerating every build 
step
that a maintainer would do. The problem here, is that a coreutils release,
for example, has modules that would not be found in a gnulib release.

So hopefully we all agree that in general this should be done,
and we're just discussing whether or not making it automatic is appropriate,
and if so, then how best to do it.

I have a v2 patch ready if the following arguments make enough sense...
or perhaps sending a patch to gnulib would be better?

On Saturday, October 12th, 2024 at 18:12, Collin Funk <collin.fu...@gmail.com> 
wrote:

>
> I disagree with the patch since it doesn't work for every package that
> uses Gnulib. Specifically, it assumes the Makefile name 'gnulib.mk'
> which is used by Coreutils.
>
> In Coreutils' bootstrap.conf file we have:
>
> gnulib_tool_option_extras="--tests-base=gnulib-tests --with-tests --symlink\
> --makefile-name=gnulib.mk --automake-subdir
> "
>

This is my fault, I was blindsided by lines in the bootstrap script
that for some reason made me think that "gnulib.mk" is the default filename...

For that reason, the patch is no good as it is,
but let's consider a possible v2 or another alternative...


On Saturday, October 12th, 2024 at 18:51, Bruno Haible <br...@clisp.org> wrote:

>
> The patch is wrong for three reasons:
>
> 1) Inversion of dependencies.
>
> Gnulib relies on Automake. (And Automake relies on Autoconf. And
> Autoconf relies on m4.) If anyone introduces a dependency in the
> opposite direction, namely that Automake relies on Gnulib (in the
> sense that Automake incorporate knowledge about how gnulib-tool
> operates), this makes evolution of both Automake and Gnulib
> extremely hard.


"dependency" is a really strong word. I would not describe this as such.

First, there already is an inversion of dependencies, in some sense,
because automake processes the output of gnulib-tool as an input.
I don't see much of a difference between automake interpreting
the output of gnulib-tool compared to the metadata in it's output.
As I see it, gnulib is well enough established at this point
that I think a large change in existing options naming or functionality is 
unlikely.

If it makes you more comfortable,
what if gnulib-tool set this with an EXTRA_DIST variable?

Second, in the way that I intend to apply this idea, each step of detection
is protected by an if-statement, and if any if-statement fails,
the default behavior is no change in functionality at all.
In case something does change, and the functionality of this change
goes from the intended action to no effect, this should be
easily discoverable by the maintainer when they attempt to run bootstrap
from a distribution tarball, as the build would fail with missing source.

I intend for automake to be able to read whatever gnulib produces
in a similar way that a human would:

  "It looks like this project had local modules,
  but I can't find them in the release,
  I wonder which directory it is in the project's VCS,
  let me look at each xyz.mk that becomes input to automake
  for the one made by gnulib-tool, and read the options."

becomes something like

  "automake finds magic string --local-dir=([^\\\s]*),
  in a file that has magic string "Generated by gnulib-tool",
  therefore there was a local gnulib module used here,
  let's distribute the info for the modules
  but only if the directory exists."

>
> It is as if the assembler had built-in knowledge of instruction
> patterns that a compiler emits. It makes it very hard for the
> compiler and for the assembler to evolve. Fortunately we don't
> see this mistake in many areas. But in areas where it exists, it
> causes major trouble, such as with the 'GNU-stack' section.
>

...maybe a better analogy would be "dependency tracking"
in which information is stored about how an object was compiled
for use after the fact...


> 2) gnulib-tool's interface with the package maintainer is that
> gnulib-tool, after copying a number of files, makes a couple (5-10)
> of instructions for the package maintainer. For example:
>
> Don't forget to
> - "include Makefile.gnulib" from within "lib/Makefile.am",
> - "include Makefile.gnulib" from within "tests/Makefile.am",
> - mention "-I gnulib-m4" in ACLOCAL_AMFLAGS in Makefile.am
> or add an AC_CONFIG_MACRO_DIRS([gnulib-m4]) invocation in ./configure.ac,
> - mention "gnulib-m4/gnulib-cache.m4" in EXTRA_DIST in Makefile.am,
> - invoke gl_EARLY in ./configure.ac, right after AC_PROG_CC,
> - invoke gl_INIT in ./configure.ac.


I was under the impression that these instructions are printed
because automake is simply not capable of making these changes for you.
This is especially true for the last 2...


>
> The reason behind this is that the package maintainer MUST remain
> the master of his build system. We do NOT want a build system in
> which the package maintainer is a victim.
>

And even with applying this idea, the maintainer is still in control
thanks to the dist-hook rule. The intention is simply to make
automake a little bit more automatic to make the maintainer's job easier,
not to sequester control...

In fact, if following the logic of the letter I linked,
it would be irresponsible for a maintainer to attempt
to block the inclusion of these files in distribution,
and whatever else automake already automatically includes
is also within the criteria that it's their responsibility to include it.


> But in this case, even this would be wrong, because:
>
> 3) The directory that is the argument of --local-dir is entirely
> maintained by the package. The developer decides which of the files
> he wants to distribute, and which he wants to keep for himself and
> not distribute. For example, it is perfectly reasonable for a
> developer to add gl/modules/<module>.diff if he wants to have
> a locally modified copy of <module> for his own purposes.
> It is not different from subdirectories with C source code, from
> subdirectories with data, from subdirectories with unit tests, etc.

Whenever I have seen a file like gl/modules/<module>.diff, it's usually
to fix a build problem that an end user or packager
would equally have problems building without,
making the file just as critical to have distributed as the source itself,
in the case that the end user or packager
needs to re-generate the pre-generated files.

If that's not the case, the maintainer should know to clear out modifications 
made
for "personal use" or testing, etc. before attempting to make a distribution.

However, if including an entire subdirectory is not acceptable, this too can be 
reworked
as the resulting Makefile.am snippet from gnulib-tool includes the list of 
modules
which can be parsed just as well as the subdirectory is.

I would argue that the "modules" files should be considered part of "the source"
just as much as the literal C source files,
as you cannot build from scratch without them,
compared to files like READMEs where whether one README should be included
vs another would be more of a open debate
since it has no effect on the ability to build with any method...

--
MCP