Am 13.03.2025 um 21:35 schrieb David Malcolm:
On Thu, 2025-03-13 at 12:11 +0100, Simon Sobisch wrote:
Thanks for your work on adding a testsuite. Can you please explain
why
you do this when a complete testsuite exists in autoconf (autotest)
format (which roots back to decade of work in GnuCOBOL, with all
copyrights for that already with the FSF)?
Is the existence of this in upstream [1] just unknown (because it was
not part of the initial patches [for reasons I not understood])?
Is the format such a big issue (note: previous discussions elaborated
"a
test suite is very important and other frontends also use a framework
other than dejagnu)?
If dejagnu is the way to go:
* Shouldn't there be deprecation of autotest in autoconf (of course
only
if that preference is also outside of gcc)?
* Shouldn't there be a (at least semi automated) script / migration
tool
(at least for this specific time in place to convert the "UAT" once
into
dejagnu format)?
Thanks for giving me some context on this,
Simon
[1]:
https://gitlab.cobolworx.com/COBOLworx/gcc-cobol/-/tree/master+cobol/gcc/cobol/UAT
Hi Simon
Does the UAT testsuite have coverage for what happens on invalid code?
For example, in
https://gcc.gnu.org/pipermail/gcc-patches/2025-March/677481.html
my patch adds test coverage for the output on one kind of typo (or, at
least, I tried to, my knowledge of COBOL is essentially 0); I put this
in Richard's DejaGnu suite since I have lots of similar tests for other
frontends.
Yes. That is what the "syn" tests are about, for example
https://gitlab.cobolworx.com/COBOLworx/gcc-cobol/-/blob/master+cobol/gcc/cobol/UAT/testsuite.src/syn_value.at
# Unexpected return code 0 failure - We don't implement
# type checking TODO
AT_DATA([prog.cob], [
IDENTIFICATION DIVISION.
PROGRAM-ID. prog.
DATA DIVISION.
WORKING-STORAGE SECTION.
* Gnu throws ERROR on next line TODO
01 X-SPACE PIC 999 VALUE SPACE.
* Gnu Throws WARNING on next two lines TODO
01 X-ABC PIC 999 VALUE "abc".
01 X-12-3 PIC 999 VALUE 12.3.
01 X-123 PIC 999 VALUE 123.
* Gnu Throws WARNING on next line
01 X-1234 PIC 999 VALUE 1234.
PROCEDURE DIVISION.
STOP RUN.
])
AT_CHECK([$COMPILE_ONLY prog.cob], [1], ,
[prog.cob:7:25: error: numeric type X-SPACE VALUE 'SPACES' requires
numeric VALUE
7 | 01 X-SPACE PIC 999 VALUE SPACE.
| ^
prog.cob:9:25: error: numeric type X-ABC VALUE 'abc' requires numeric VALUE
9 | 01 X-ABC PIC 999 VALUE "abc".
| ^
prog.cob:10:25: error: integer type X-12-3 VALUE '12.3' requires integer
VALUE
10 | 01 X-12-3 PIC 999 VALUE 12.3.
| ^
prog.cob:13:25: error: numeric X-1234 VALUE '1234' holds only 3 digits
13 | 01 X-1234 PIC 999 VALUE 1234.
| ^
cobol1: error: failed compiling prog.cob
])
Notes:
* in GnuCOBOL we have specific tests that check the "extended" format
with context; in all other cases we pass (as part of the $COMPILE
above) -fdiagnostics-plain-output - because we only want the user
message, not the formatting tested overall - and producing and
comparing less output also saves some (minor) computation
* I'd definitely suggest that UAT is adjusted similar before the
conversion for the same reasons
* as hinted in the UAT notes above, GnuCOBOL test results are different,
here the original result from the file this test in UAT was based on:
prog.cob:6: error: invalid VALUE clause
prog.cob:7: warning: numeric value is expected
prog.cob:8: warning: value size exceeds data size
prog.cob:10: warning: value size exceeds data size
[note: according to ISO those should all be errors, but other COBOL
implementations don't care and truncate "per MOVE rules", so cobc's
default is a warning here]
Just for your amusement, that's GC 3.2's output (with -Wall)
prog.cob:6: error: invalid VALUE clause
4 | DATA DIVISION.
5 | WORKING-STORAGE SECTION.
6 > 01 X-SPACE PIC 999 VALUE SPACE.
7 | 01 X-ABC PIC 999 VALUE "abc".
8 | 01 X-12-3 PIC 999 VALUE 12.3.
prog.cob:7: warning: numeric value is expected [-Wothers]
5 | WORKING-STORAGE SECTION.
6 | 01 X-SPACE PIC 999 VALUE SPACE.
7 > 01 X-ABC PIC 999 VALUE "abc".
8 | 01 X-12-3 PIC 999 VALUE 12.3.
9 | 01 X-123 PIC 999 VALUE 123.
prog.cob:8: warning: value size exceeds data size [-Wothers]
6 | 01 X-SPACE PIC 999 VALUE SPACE.
7 | 01 X-ABC PIC 999 VALUE "abc".
8 > 01 X-12-3 PIC 999 VALUE 12.3.
9 | 01 X-123 PIC 999 VALUE 123.
10 | 01 X-1234 PIC 999 VALUE 1234.
prog.cob:10: warning: value size exceeds data size [-Wothers]
8 | 01 X-12-3 PIC 999 VALUE 12.3.
9 | 01 X-123 PIC 999 VALUE 123.
10 > 01 X-1234 PIC 999 VALUE 1234.
11 | PROCEDURE DIVISION.
12 | STOP RUN.
Having a good user experience on incorrect code is important, so we
need some kind of test coverage for this. The new DejaGnu-based tests
seem to work well for that (as per the patch). I don't know if this is
something that could/should be shared with GnuCOBOL, given that there
might well be differences in error-handling behavior between GCC COBOL
and GnuCOBOL.
I totally agree for the user need of good experience on incorrect code
(and libgdiagnostic will likely help - possibly also GnuCOBOL users - to
have some nice features that GCC does without reimplementing everything).
Testing for that is important for many reasons (one is: you want to
ensure the compiler does not crash) - and that test should include the
user-visible part, which should "work" also without context (the gcobol
ones above do, the cobc ones doesn't).
But as gcobol's and GnuCOBOL's parser and frontend is quite different, I
don't see a way to have an easy common testbed.
One option that would be marvelous would be a test checking both cobc's
and gcobol's messages (and with the use of gcobc it could also use the
same invocation) - but as noted that would have to include two
"conditional" expected outputs. So far no one jumped and said "here, I
wanna do that work"...
Thoughts?
Dave
Simon
