Re: recursion limit of 1024 exceeded

[Bruno Haible]

Dagobert Michelsen wrote:
> autoconf (GNU Autoconf) 2.69  
> automake (GNU automake) 1.16.1  

Even with these (not exactly the newest) versions of autoconf and
automake, the gnulib-tool create-testdir command succeeds for me.

> meanwhile I tried to update autoconf to 2.70 with
> 3 unexpected failures and automake to 14 unexpected failures which I will
> report on the respective mailing lists.

Yes, please!

> >> gm4:configure.ac:4574: recursion limit of 1024 exceeded, use -L to 
> >> change it

The maximum recursion limit of m4 depends how it was built. Mine has

  $ m4 --help|grep limit
-L, --nesting-limit=NUMBER   change nesting limit, 0 for unlimited [0]

Yours surely has

  $ m4 --help|grep limit
-L, --nesting-limit=NUMBER   change nesting limit, 0 for unlimited [1024]

The default value for -L comes from this code:

  if (c_stack_action (fault_handler) == 0)
nesting_limit = 0;

This means, if m4 can react on stack overflow without knowing a priori how many
recursive macro invocations it can handle — this is the case when libsigsegv is
installed or the simple heuristic in c-stack.c works —, it does so. Only in the
other case, it uses the fixed-size nesting limit.

What I don't understand, is:
  - why on your system the c-stack.c heuristic, specifically tailored for
Solaris, did not work,
  - why, when (on a glibc system) I use a limit even as low as -L 1,
'aclocal -I glm4' and 'autoconf' run fine in the generated testdir.

> > m4 --version  
> m4 (GNU M4) 1.4.18  

By the way, autom4te does not look up m4 in $PATH; it uses the M4 environment
variable or a program in a fixed location. Look into autom4te to find out which
m4 it actually uses.

Bruno

Re: recursion limit of 1024 exceeded

[Dagobert Michelsen]

Hi Bruno,

Am 12.12.2020 um 17:25 schrieb Bruno Haible :
 gm4:configure.ac:4574: recursion limit of 1024 exceeded, use -L to 
 change it
> 
> The maximum recursion limit of m4 depends how it was built. Mine has
> 
>  $ m4 --help|grep limit
>-L, --nesting-limit=NUMBER   change nesting limit, 0 for unlimited [0]
> 
> Yours surely has
> 
>  $ m4 --help|grep limit
>-L, --nesting-limit=NUMBER   change nesting limit, 0 for unlimited [1024]

Nope, same as yours:

dam@experimental10x [global]:/home/dam > m4 --help | grep limit 
  -L, --nesting-limit=NUMBER   change nesting limit, 0 for unlimited [0]

> This means, if m4 can react on stack overflow without knowing a priori how 
> many
> recursive macro invocations it can handle — this is the case when libsigsegv 
> is
> installed or the simple heuristic in c-stack.c works —, it does so. Only in 
> the
> other case, it uses the fixed-size nesting limit.

Does this mean that m4 should also link against libsigsegv? This is not the 
case:

dam@experimental10x [global]:/home/dam > ldd /opt/csw/gnu/m4 
libc.so.1 => /lib/libc.so.1
libm.so.2 => /lib/libm.so.2

> What I don't understand, is:
>  - why on your system the c-stack.c heuristic, specifically tailored for
>Solaris, did not work,
>  - why, when (on a glibc system) I use a limit even as low as -L 1,
>'aclocal -I glm4' and 'autoconf' run fine in the generated testdir.

I see. Is there anything I can do to further drilldown the issue?

>>> m4 --version  
>> m4 (GNU M4) 1.4.18  
> 
> By the way, autom4te does not look up m4 in $PATH; it uses the M4 environment
> variable or a program in a fixed location. Look into autom4te to find out 
> which
> m4 it actually uses.

It is my OpenCSW m4 1.4.18 as reported:

my $m4 = $ENV{"M4"} || '/opt/csw/bin/gm4‘;


Best regards

  — Dago

-- 
"You don't become great by trying to be great, you become great by wanting to 
do something,
and then doing it so hard that you become great in the process." - xkcd #896

Re: recursion limit of 1024 exceeded

[Dagobert Michelsen]

Hi Bruno,

Am 11.12.2020 um 17:01 schrieb Bruno Haible :
>> dam@experimental10x [global]:/home/dam/work/gnulib > PATH=/opt/csw/gnu:$PATH 
>> ./gnulib-tool --create-testdir --dir=./testdir --single-configure 
>> --without-privileged-tests
> 
> On a glibc system, this command line works fine. It takes 1 or 2 hours,
> though.

Yes, similar on Solaris.

>> executing aclocal -I glm4
>> gm4:configure.ac:4574: recursion limit of 1024 exceeded, use -L to change 
>> it
> 
> You gave `m4 --version`. But what is `autoconf --version` and
> `automake --version`?

Sure:

dam@experimental10x [global]:/home/dam/work/gnulib > m4 --version  
m4 (GNU M4) 1.4.18  
Copyright (C) 2016 Free Software Foundation, Inc.  
License GPLv3+: GNU GPL version 3 or later .  
This is free software: you are free to change and redistribute it.  
There is NO WARRANTY, to the extent permitted by law.  
  
Written by Rene' Seindal.  
dam@experimental10x [global]:/home/dam/work/gnulib > autoconf --version  
autoconf (GNU Autoconf) 2.69  
Copyright (C) 2012 Free Software Foundation, Inc.  
License GPLv3+/Autoconf: GNU GPL version 3 or later  
,   
This is free software: you are free to change and redistribute it.  
There is NO WARRANTY, to the extent permitted by law.  
  
Written by David J. MacKenzie and Akim Demaille.  
dam@experimental10x [global]:/home/dam/work/gnulib > automake --version  
automake (GNU automake) 1.16.1  
Copyright (C) 2018 Free Software Foundation, Inc.  
License GPLv2+: GNU GPL version 2 or later 
  
This is free software: you are free to change and redistribute it.  


m4 should be current, meanwhile I tried to update autoconf to 2.70 with
3 unexpected failures and automake to 14 unexpected failures which I will
report on the respective mailing lists.
Do you think the error could be from outdated versions of these two?


Best regards

  — Dago

-- 
"You don't become great by trying to be great, you become great by wanting to 
do something,
and then doing it so hard that you become great in the process." - xkcd #896

Re: recursion limit of 1024 exceeded

[Paul Eggert]

On 12/12/20 11:40 AM, Dagobert Michelsen wrote:

I suggest running './gnulib-tool --create-testdir --dir FOO ...' on a more up-to-date 
platform, copying FOO to Solaris 10, and then testing just './configure && 
make' on the copy of FOO on Solaris 10.



I see, would that also count as viable solution for Gnulib CI on Solaris?


Yes, it'd be a viable test of the part of Gnulib that we care about for 
Solaris 10.


Of course, you're welcome to continue doing an entire port the whole shebang 
(GNU m4, Autoconf, Automake etc.) to Solaris 10 and it sounds like you're 
doing that anyway and so might want to look into whatever problem you're 
having there. But if the idea is merely to test Gnulib on Solaris 10, 
testing the output of gnulib-tool should be enough.

Re: recursion limit of 1024 exceeded

[Paul Eggert]

On 12/12/20 4:13 AM, Dagobert Michelsen wrote:

m4 should be current, meanwhile I tried to update autoconf to 2.70 with
3 unexpected failures and automake to 14 unexpected failures which I will
report on the respective mailing lists.
Do you think the error could be from outdated versions of these two?


It's possible, since I just installed some patches into Gnulib related to 
Autoconf 2.70. I did test those patches with Autoconf 2.69 too (as patched 
for Fedora 33), so if the problem is Autoconf related, most likely it's a 
problem fixed in the Fedora patches for Autoconf 2.69 which means you would 
be better off with Autoconf 2.70.


Despite my fondness and nostalgia for Solaris, I don't bother with testing 
gnulib-tool on Solaris 10 because Solaris 10 ships with GNU m4 1.4.2, 
Autoconf 2.59, and Automake 1.8.3, and so it's no longer a suitable porting 
target for Gnulib development. That is, although you can build from 
Gnulib-based tarballs on Solaris 10, you can't build from Git development 
trees (Solaris 10 doesn't even have Git) without doing a lot of work and 
it's not worth the effort to me. Instead, I suggest running './gnulib-tool 
--create-testdir --dir FOO ...' on a more up-to-date platform, copying FOO 
to Solaris 10, and then testing just './configure && make' on the copy of 
FOO on Solaris 10.

Re: recursion limit of 1024 exceeded

[Dagobert Michelsen]

Hi Paul,

Am 12.12.2020 um 20:29 schrieb Paul Eggert :
> On 12/12/20 4:13 AM, Dagobert Michelsen wrote:
>> m4 should be current, meanwhile I tried to update autoconf to 2.70 with
>> 3 unexpected failures and automake to 14 unexpected failures which I will
>> report on the respective mailing lists.
>> Do you think the error could be from outdated versions of these two?
> 
> It's possible, since I just installed some patches into Gnulib related to 
> Autoconf 2.70. I did test those patches with Autoconf 2.69 too (as patched 
> for Fedora 33), so if the problem is Autoconf related, most likely it's a 
> problem fixed in the Fedora patches for Autoconf 2.69 which means you would 
> be better off with Autoconf 2.70.
> 
> Despite my fondness and nostalgia for Solaris, I don't bother with testing 
> gnulib-tool on Solaris 10 because Solaris 10 ships with GNU m4 1.4.2, 
> Autoconf 2.59, and Automake 1.8.3, and so it's no longer a suitable porting 
> target for Gnulib development. That is, although you can build from 
> Gnulib-based tarballs on Solaris 10, you can't build from Git development 
> trees (Solaris 10 doesn't even have Git) without doing a lot of work and it's 
> not worth the effort to me. Instead, I suggest running './gnulib-tool 
> --create-testdir --dir FOO ...' on a more up-to-date platform, copying FOO to 
> Solaris 10, and then testing just './configure && make' on the copy of FOO on 
> Solaris 10.

I see, would that also count as viable solution for Gnulib CI on Solaris? I 
have other,
more recent systems around to use on the CI system of that would help.


Best regards

  — Dago

-- 
"You don't become great by trying to be great, you become great by wanting to 
do something,
and then doing it so hard that you become great in the process." - xkcd #896

Fix gnulib-tool error when some modules occur in tests/

[Bruno Haible]

Last week, the m4 continuous integration failed:

$ ./bootstrap --skip-git --gnulib-srcdir="$GNULIB_SRCDIR"
...
bootstrap: running: autoreconf --symlink --install
missing file tests/fopen.c
configure.ac:150: error: expected source file, required through AC_LIBSOURCES, 
not found
m4/gnulib-comp.m4:669: M4_INIT is expanded from...
configure.ac:150: the top level
autom4te: m4 failed with exit status: 1
aclocal: error: autom4te failed with exit status: 1
autoreconf: aclocal failed with exit status: 1

What happened? Through some new module dependencies, the module 'fopen-gnu'
was added to the tests modules. And the main modules, used for the 'm4'
program, continued to contain the 'fopen' module. Now, 'fopen-gnu' depends
on 'fopen', and the 'fopen' module contains the file lib/fopen.c. We thought
that was sufficient for the AC_LIBOBJ([fopen]) invocations in the 'fopen'
and 'fopen-gnu' modules. It isn't.

We have a number of such situations like
  fopen, fopen-gnu
  malloc-posix, malloc-gnu
  strstr-simple, strstr
  getopt-posix, getopt-gnu

How should such situations be handled? The answer is:

1) Since 'fopen-gnu' contains an AC_LIBOBJ([fopen]) invocation, it MUST
   list lib/fopen.c among its file list. This is needed for the cited
   case, where the package contains
 lib/fopen.c (for the sake of the 'fopen' module), and
 tests/fopen.c (for the same of the 'fopen-gnu' module).

   Trying to get rid of one of the two files would be quite complex.
   Say, we wanted to keep lib/fopen.c and not keep tests/fopen.c.
   Then, either the lib/Makefile.am would have to be informed whether
   to compile lib/fopen.c into lib/fopen.o purely for the tests (oh oh),
   or the tests/Makefile.am would compile lib/fopen.c into tests/fopen.o
   but only if lib/Makefile.am has not already created lib/fopen.o (oh oh
   as well). IMO, this does not lead to a maintainable solution.

   If the autoconf tests for the 'fopen-gnu' module set some flags that
   make the lib/fopen.c assume the GNU behaviour, that is OK if 'fopen-gnu'
   does not have additional dependencies that might cause link errors in
   the lib/ directory (*).

   The lib/ directory and the tests/ directory will then have (possibly
   different) fopen.o files, which each define an 'rpl_fopen' symbol.
   The tests will see the one from tests/fopen.o, since the tests are
   linked with
  libtests.a ../lib/libgnu.a libtests.a ../lib/libgnu.a ...
   Also, shared libraries are not a problem, since libtests.a is always
   a static library, and when a test is linked with
  libtests.a ../lib/libgnu.la libtests.a ../lib/libgnu.la ...
   the tests/fopen.o ends up in the executable, and when referenced from
   the executable, symbols in the executable have precedence over symbols
   in shared libraries (not only on ELF systems, but also on other
   platforms: macOS, AIX, Windows).

(*) These link errors could be resolved by moving the dependency down from
the 'fopen-gnu' module to the 'fopen' module.

2) The getopt-posix, getopt-gnu use a different technique: when there
   is a request for module 'getopt-posix' and a request for module
   'getopt-gnu' in the scope of the same configure file, they are treated
   as if both had requested 'getopt-gnu'.

   This approach has the advantage that only one AC_LIBOBJ([getopt]) exists
   (among lib/ and tests/ of the same gnulib-tool invocation). It saves
   code. But there is quite some boilerplate macrology. At this point,
   I shy away from duplicating this macrology into a dozen of other modules.

3) There are also a number of modules that have AC_LIBOBJ invocations for
   shared data:
   AC_LIBOBJ([mbsrtowcs-state])
   AC_LIBOBJ([wcsrtombs-state])
   AC_LIBOBJ([c32srtombs-state])
   AC_LIBOBJ([mbsrtoc32s-state])
   AC_LIBOBJ([expl-table])
   AC_LIBOBJ([lc-charset-dispatch])
   AC_LIBOBJ([sincosl])
   AC_LIBOBJ([trigl])
   AC_LIBOBJ([stat-w32])
   For these cases, if approach 1) does not produce satisfactory results,
   it would be possible to move each of these .c files into a separate
   module.

All in all, the following patch seems sufficient for me at this point.


2020-12-12  Bruno Haible  

Fix gnulib-tool error when some modules occur in tests/.
* doc/gnulib.texi (Specification): Update statistics.
(Autoconf macros): Don't suggest to use AC_LIBOBJ in a .m4 file.
(Using AC_LIBOBJ): New section.
* check-AC_LIBOBJ: New file.
* modules/fnmatch-gnu (Files): Add lib/fnmatch.c.
* modules/fopen-gnu (Files): Add lib/fopen.c.
* modules/memmem (Files): Add lib/memmem.c.
* modules/renameat (Files): Add lib/at-func2.c.
* modules/strcasestr (Files): Add lib/strcasestr.c.
* modules/strstr (Files): Add lib/strstr.c.

diff --git a/doc/gnulib.texi b/doc/gnulib.texi
index bddb806..d646d6d 100644
--- a/doc/gnulib.texi
+++ b/doc/gnulib.texi
@@ -166,6 +166,7 @@ consistency with gnulib.
 * Specification::