On 2015-06-02 09:40 -0700, Arthur Schwarz wrote: > [Nick Bowler wrote] > The plan line has to be mandatory for this feature to work as > intended (i.e., for the TAP consumer to determine whether a > producer has run to completion or not). An optional plan would > be useless. > > Sorry to be such a pest. > > Your statement that "... determine whether a producer has run to completion > or not" is puzzling. The TAP Standard specifically states that if the test > plan is 1..N and the number of test lines are k < N, then the k+1 .. N > missing test lines are to be considered as skipped.
Where does it say that? It is not correct to treat missing test results as skips. From http://testanything.org/tap-version-13-specification.html Skipping a few The following listing plans on running 5 tests. However, our program decided to not run tests 2 thru 5 at all. To properly report this, the tests are marked as being skipped. TAP version 13 1..5 ok 1 - approved operating system # $^0 is solaris ok 2 - # SKIP no /sys directory ok 3 - # SKIP no /sys directory ok 4 - # SKIP no /sys directory ok 5 - # SKIP no /sys directory It is an error for the number of actual test results to differ from the plan. This is the whole point of the plan: the consumer always knows how many test results are expected. This enables the consumer to tell the difference between a producer that crashed before completing all the tests, and a producer that ran normally. See above. > There is one issue not addressed in the TAP Standard and not addressed in > the Automake Manual. What happens if k > N? If k > N, then the number of test cases does not match the plan, so this is an error. TAP allows k < N (see above). The question of what happens if k > N seems reasonable. As an extension, the Automake Manual does not state what will happen if k > N (nor if k < N). This also reflects of the TAP Protocol statement that the test plan can be at the beginning or end of the test. The issue is easy to address if the test plan is at the beginning of the document but will require that an implementation use deferred test line posting if the test plan is at the end of the document. I think that the TAP Protocol is poor in this regard. art Regards, -- Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
