Hello list,
I have been testing gnulib on IBM z/OS in recent days, and have found
a few compatibility issues (some old, some new) that I'd like to
address here.
This is intended to be a comprehensive review, so it will be a little
long; please bear with me!
BROKEN LOCALE FUNCTIONS
The configure script finds the following:
checking for xlocale.h... no
checking for duplocale... yes
checking for uselocale... yes
checking for newlocale... yes
checking for freelocale... yes
checking whether locale.h conforms to POSIX:2001... yes
checking whether uselocale works... no
checking whether duplocale(LC_GLOBAL_LOCALE) works... no
checking whether setlocale supports the C locale... yes
The problem is, those duplocale(), newlocale(), and freelocale()
functions are not usable. Not only are they not declared in locale.h,
not only does the runtime crash if you call them, the locale_t type
isn't even defined. Here is a portion of the config.log output for the
"duplocale works" test:
configure:37076: checking whether duplocale(LC_GLOBAL_LOCALE) works
configure:37166: xlc-wrap -o conftest -g -qfloat=ieee -qlanglvl=extc99
-qenumsize=4 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39
-qhaltonmsg=CCN3296 conftest.c >&5
ERROR CCN3275 ./conftest.c:405 Unexpected text loc encountered.
ERROR CCN3045 ./conftest.c:405 Undeclared identifier locale_t.
ERROR CCN3045 ./conftest.c:415 Undeclared identifier LC_GLOBAL_LOCALE.
ERROR CCN3045 ./conftest.c:415 Undeclared identifier loc.
CCN0793(I) Compilation failed for file ./conftest.c. Object file not
created.
configure:37166: $? = 12
configure: program exited with status 12
And the "uselocale works" test, which fails in the crashing way:
configure:24730: checking whether uselocale works
configure:24757: xlc-wrap -o conftest -g -qfloat=ieee -qlanglvl=extc99
-qenumsize=4 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39
-qhaltonmsg=CCN3296 conftest.c >&5
configure:24757: $? = 0
configure:24757: ./conftest
CEE3728S The use of a function, which is not supported by this release of
Language Environment was detected.
From compile unit /tmp/gnulib-build/conftest.c at entry point main
at statement 275 at compile unit offset
+00000074 at entry offset +00000074 at address 2190A9BC.
configure:24757: $? = 137
configure: program exited with status 137
Currently, gnulib reads this as "Oh, the system duplocale() et al. are
broken, let me replace them." Unfortunately, this results in build
errors at the replacement function prototype, due to the missing
locale_t type:
xlc-wrap -DHAVE_CONFIG_H -I. -I/u/username/testdir/gllib -I..
-DGNULIB_STRICT_CHECKING=1 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39
-qhaltonmsg=CCN3296 -g -qfloat=ieee -qlanglvl=extc99 -qenumsize=4 -c -o
hard-locale.o /u/username/testdir/gllib/hard-locale.c
ERROR CCN3166 ./locale.h:702 Definition of function locale_t requires
parentheses.
ERROR CCN3276 ./locale.h:702 Syntax error: possible missing '{'?
In order to get a successful build, I have to set
ac_cv_func_duplocale=no
ac_cv_func_newlocale=no
before configuring. Could these functions (perhaps freelocale() too) be
treated as non-existent if there is no usable locale_t type?
Alternately, checking for their (missing) declarations should also work.
PTHREAD ENVIRONMENT
z/OS has effectively two major pthread interfaces: _OPEN_THREADS, and
_UNIX95_THREADS. The latter is the one you want to use, because its API
is compatible with other systems. (_OPEN_THREADS uses e.g. a different
signature for pthread_getspecific() that comes from a draft POSIX
specification; refer to
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/ptgetsp.htm
for the gory details. pthread_detach() is different, too, and
pthread_cond_timedwait() can return EAGAIN.)
Additional information on z/OS feature test macros, if desired:
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/ftms.htm
More specifically, to get a suitable pthreads interface, you need to
#define _UNIX95_THREADS and _XOPEN_SOURCE=600. (This could perhaps be
added to AC_USE_SYSTEM_EXTENSIONS for this system.)
However, one annoyance of _UNIX95_THREADS is that it does not define
PTHREAD_RWLOCK_INITIALIZER, supposedly because the SUSv3 standard (which
is what that feature test macro requests) does not specify it. However,
IBM does provide the "implementation-defined"
PTHREAD_RWLOCK_INITIALIZER_NP, which can be used in its place. See the
note here:
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/rp0r0i.htm
I would thus suggest adding something like the following to
lib/pthread.in.h:
#if defined(__MVS__) && !defined(PTHREAD_RWLOCK_INITIALIZER)
# define PTHREAD_RWLOCK_INITIALIZER PTHREAD_RWLOCK_INITIALIZER_NP
#endif
SHELL SCRIPTING SNAFUS
In the z/OS environment, if you want to use a proper Bash shell, you
typically get the build provided by Rocket Software:
https://www.rocketsoftware.com/product-categories/mainframe/bash-zos
The problem, however, is that shell scripting with this version of Bash
can be a little tricky, because it doesn't fully embrace EBCDIC. A brief
demonstration:
bash-4.3$ echo $BASH_VERSION
4.3.46(51)-release
bash-4.3$ printf ABC | od -t x1
0000000000 41 42 43
0000000003
bash-4.3$ /bin/printf ABC | od -t x1
0000000000 C1 C2 C3
0000000003
bash-4.3$ echo ABC | grep ABC
(no output)
bash-4.3$ /bin/echo ABC | grep ABC
ABC
bash-4.3$ echo ABC >test.txt
bash-4.3$ cat test.txt
????
bash-4.3$ od -t x1 <test.txt
0000000000 41 42 43 0A
0000000004
bash-4.3$ echo `cat test.txt`
ABC
It's very confusing and annoying, and when I first tested gnulib using
this version of Bash, it ended up looping infinitely on running
test-atexit.sh. The loop occurred in mktempd_(), in gltests/init.sh,
because the sheer brokenness of the shell environment resulted in the
MAX_TRIES_ logic not working.
In order to get the shell to work sanely, you have to set a bevy of
environment variables discussed in
/usr/lpp/ported/bash-4.3/share/doc/bash/4.3/README.ZOS
(I cannot find a public copy to link to, unfortunately). These are the
settings I used which allow things to work:
_ENCODE_FILE_NEW=IBM-1047
_ENCODE_FILE_EXISTING=IBM-1047
_CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
_BPXK_AUTOCVT=ON
_TAG_REDIR_ERR=txt
_TAG_REDIR_IN=txt
_TAG_REDIR_OUT=txt
(Note: This may not be a minimal set)
While I would not recommend giving to init.sh knowledge of the above
variables, I think it would be helpful to do some basic sanity checking
(like the echo|grep invocation above) to avoid more pathological
breakage later in the script. The failure message could include a hint
to the user about what's wrong with the shell, and what needs to be done
to fix it.
Note: The Rocket Software version of Bash is not the only one that
exists on this platform. My org has an older version, apparently
compiled by us long ago, that has given me a lot fewer headaches.
Therefore, no assumption can be made about Bash on z/OS being the Rocket
version. Unfortunately, as far as I can tell, there is no good way of
uniquely identifying the Rocket version, either (as you can see, the
BASH_VERSION string has not been tweaked).
ENVIRONMENT VARIABLES
One annoyance on this platform is that the default behavior of things
can sometimes be weirdly different from other platforms, in a way which
breaks programs expecting normal Unix/POSIX behavior. Unfortunately,
IBM's response to this is often not "Let us fix that so it works like
other Unix systems," but "That's too bad. We can't change the default
behavior because mumblemumble, but we can provide an environment
variable that, if set, with cause that thing to behave in the manner
you expect."
At present, the only such variable worth mentioning here is
_EDC_SIG_DFLT, which when set to 1, causes certain default signal
handlers *not* to print out messages:
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.cbcpx01/edc_sig_dflt.htm
This is something that should at least be set in the gnulib test
environment so that test-sigpipe.sh doesn't break, but may be worth
adding to the library itself so that programs can continue to expect
"normal" signal semantics.
(There are other environment variables documented at the link above that
may be of interest, but I have not found any additional ones to be
necessary to fix issues in gnulib's test suite.)
AUTOCONF ISSUES
A few issues in this environment relate to compiler quirks, which I
intend to bring to the attention of the Autoconf folks:
* -qfloat=ieee is needed to use "normal" IEEE754 floating-point format
(the default is IBM's proprietary "hexadecimal" format)
* -qhaltonmsg=CCN3296 is needed so that the compiler treats missing
header files as an error instead of a warning (!)
* Some extra rigmarole is needed so that a system foo.h is not favored
over a foo.h file in some -I<dir> location; the "xlc-wrap" script I am
using addresses this, and is as follows:
#!/bin/sh
exec xlc -qnosearch "$@" -qsearch=/usr/include
* Other compiler frontends are available on z/OS: c89, c99, cc, xlC,
xlc++, xlclang, xlclang++. Not all of them are usable, and some of
them take a strange option syntax; generally the ones starting with
"xl" are easier to deal with
(Some work previously done on this can be seen in GNU Gawk's
m4/arch.m4 file)
MISCELLANEOUS BUGS
There are a handful of additional issues that I would like to mention
here for completeness, but am taking up primarily with IBM, rather than
this list. If anyone would like me to elaborate on any of these, please
feel free to ask:
* [sys/]signal.h does not #define NSIG (nor SIGMAX)
* File descriptor passed to fdopendir() no longer behaves like a normal
fd; in particular, dup2(fd,fd) and fcntl(fd,F_GETFL) both fail (but
close(fd) does not), causing test-fdopendir to fail spuriously (breaks
the "fdopendir should not close fd" assertion)
* select() on /dev/null always returns 0, even though reading or writing
to it never blocks
* IBM XLC compiler does not support the C11 _Alignas() specifier, nor
UTF-8 literal strings, despite ostensibly supporting C11 (other
features are mostly there, save for the next point)
* C11 _Thread_local support is utterly broken (test-thread_local fails)
* The z/OS Make utility is severely broken: does not support VPATH, nor
recipe lines preceded with a hyphen, nor $(VAR:=.suffix) (can change a
suffix but not add a new one); thankfully I have GNU Make here
That is basically everything I have. I'd like to get this knowledge out
into the open, and hopefully integrated into gnulib and elsewhere, so
that less manual intervention is needed when building GNU software on
this platform. I am happy to answer questions, and help with testing any
proposed changes.
--Daniel
--
Daniel Richard G. || sk...@iskunk.org
My ASCII-art .sig got a bad case of Times New Roman.
