Am 07.04.2025 um 09:36 schrieb Jakub Jelinek:
On Mon, Apr 07, 2025 at 09:30:59AM +0200, Richard Biener wrote:
On Mon, Apr 7, 2025 at 9:00 AM Simon Sobisch <simonsobi...@gnu.org> wrote:
My question stands on integrating COBOLworx' UAT as-is for now
(Copyright is all on FSF; built automatically [it is autoconf, which is
a requirement for VCS checkouts], possibly also hooked into the current
test target) - with the goal to get rid of UAT later (next GCC version,
not GCC 15).
There's also the question about integrating NIST into GCC upstream -
that is a subfolder and would only be executed upon explicit call by
maintainers (newcob.val / newcob.val.gz may be either included in VCS or
even downloaded manually...).
As I repeatedly said I'd welcome a test harness like Ada ACATS for running
the NIST testsuite plus a contrib/download_cobol_nist script that downloads
the NIST file and prepares it for use. I'd suggest to, similar as with ACATS,
have a separate make target for testing (but still invoked with make check,
when present).
But it would be much better if the harness for NIST testing was in dejagnu
rather than anything else, only that can handle easily cross-compilations
with target boards, parallelization respecting make job reserve, seamless
result integration.
My understanding has been NIST is a single file from which some tool needs
to dig up individual testcases (tcl string support should be able to deal
with that), figuring out what options etc. to pass search test and from
somewhere find out the expected output for each test.
Jakub
The source is available as "newcob.val".
From there a COBOL program EXEC85 is extracted, which is then to be
compiled.
This extracted COBOL program is to be run to extract the requested NIST
modules for test with the given configuration that it reads from a file.
These modules contain more COBOL sources which are to be compiled and
run, which produce a test report each.
A final step is then to compare the results with the expectation.
I don't know enough about dejagnu to say if/how this may do several
parts (it should start to compile EXEC85 and then use that for its
configurations to extract the modules and run those - or you just
compile that and run it for extraction once - via make - then run the
modules multiple times using dejagnu).
The GnuCOBOL driver for NIST is written in perl, it "only" handles the
part of running the module tests (starting from compilation, partially
with different options as requested for some modules [like the DB one
gcc cobol does not test currently], executes them, partially checks
compiler messages - where the compiler's ability to "flag" something is
to be tested, checks the log files to gather the number of single tests
within each program and counting pass/fail [or compile error up front],
recording the execution time, and finally creates an "overview"
report.log and execution time for each module, which is then finally
compared with plain diff + make).
As I understood it, COBOLworx' has put this all into make (and dropped
part of the things that the gnucobol driver does).
Clone or look at
https://gitlab.cobolworx.com/COBOLworx/gcc-cobol/-/tree/parser/gcc/cobol/nist
to find out what COBOLworx did exactly and
https://sourceforge.net/p/gnucobol/code/HEAD/tree/branches/gnucobol-3.x/tests/cobol85/
[1] to find out what GC does.
Simon
[1] or
https://github.com/OCamlPro/gnucobol/tree/gcos4gnucobol-3.x/tests/cobol85,
if you prefer a git mirror
