Hello automakers.
Currently, the great majority of testcases in the automake testsuite
have names that give very little information about the object of the
test. An extreme example of this is the set of 55 tests on automake
conditionals, of which 44 are named along the scheme "condN.test"
(for an integer N).
This unclear naming scheme has at least two drawbacks:
1. It makes more difficult to easily find out wich are the tests
concerned with a particular feature or bug; this is especially
annoying in cases where the checks on said feature needs to be
extended or improved somehow, and it risks to cause unnecessary
repetition and duplication of tests.
2. It makes more difficult to avoid the clashing of test names in
different branches. For example, let's say we have two public
branches B1 and B2, which are managed respectively by developers
Alice and Bob, and which add respectively the features "foo" and
"bar" to automake conditionals. The conscientious developer Alice
adds a new couple of tests for her feature, and, faithful to the
current testsuite naming convention, calls them `cond51.test' and
`cond52.test'. Similarly, Bob adds a test for his feature, and
calls it `cond51.test'. Obviously, when B1 and B2 are merged to
master, a spurious conflict over `cond51.test' arises. And in
case B1 and B2 are long-lived branches, this spurious conflict
might resurface again and again with following merges.
The only advantage of the present naming scheme for tests is an
enhanced portability to DJGPP (which only supports 8+3 DOS-like
file names, thus rendering more difficult to have non-clashing
expressive names for tests); but does this matter anymore? After
all, some test names are already incompatible with DJGPP, but no
bug report about this has ever showed up. Also, if it really,
*really* should prove necessary to support DJGPP, we could setup
a dist-hook to rename the tests before distribution.
So here is my proposal: what about creating a new public branch
"tests-rename" dedicated *only* to the renaming of tests? With
this I mean that the commits going into that branch would only:
1. rename existing tests ("git mv clean2.test cygnus-distclean.test")
2. adjust references to the renamed tests (e.g. in sister tests which
refers to each other in their heading comments);
3. adjust TESTS and XFAIL_TESTS in Makefile.am accordingly.
To keep this branch in sync with the automake developement, master
would be regularly merged into it.
As Ralf already noted, the real problem with this idea is that it
might render the comparison between new and older tests result more
difficult. About this, I propose two solutions:
1. Don't merge the "tests-rename" branch into master until *after*
the next automake release. This would reduce the impact of the
problem, but obviously wouldn't completely solve it.
2. In addition to [1], set up in the "tests-rename" branch new
helper script(s) (or makefile target(s), or whatever) that
would translate the new results back to the older test names.
Opinions?
Thanks,
Stefano
