* Stefano Lattarini wrote on Tue, Jul 19, 2011 at 11:38:18AM CEST: > On Monday 18 July 2011, Ralf Wildenhues wrote: > > * Stefano Lattarini wrote on Fri, Jul 15, 2011 at 12:37:07AM CEST: > > > + - The parallel-tests harness has a new test result 'ERROR', that can be > > > + used to signal e.g., unexpected or internal errors, or failures to > > > set > > > + up test case scenarios. > > > > The NEWS entry could be a bit more precise in that the ERROR state was > > actually used before already with exit status 99, just that it is named > > ERROR now. > > > What about this? > > - Test scripts that exit with status 99 to signal an "hard error" (e.g., > and unexpected or internal error, or a failure to set up the test case > scenario) have their outcome reported as an 'ERROR' now. Previous > versions of automake reported such an outcome as a 'FAIL' (the only > difference with normal failures being that hard errors were counted > as failures even when the test originating them was listed in > XFAIL_TESTS).
Cool. > > (Please check whether the semantics of 99 already were in a stable release, > > > Well, an exit status of `99' has been triggering an hard error since at > least automake 1.11 (assuming `parallel-tests' was in use). Is this what > you were asking, Yup, thanks. > or have I misunderstood? > > A description of ERROR semantics needs to be part of automake.texi as > > well. > > > Doing that properly will require a small reorganization of the "Support > for test suites" chapter, so I'd rather do that in a follow-up patch > (which I will start working on shortly). OK. Just so we don't forget. > > Also, are you going to followup with Autoconf to rename Autotest's hard > > failure state ERROR as well? We should agree on common naming and > > semantics, so the two systems are not harder to learn than necessary. > > > Good idea, consistency is always nice. Thanks. > > I'm OK with this change once these issues are resolved. > > > > As a minor detail however, please remove the '====' greps, I've already > > mentioned why I consider '=' in the output not a good idea. > > > But those greps are there to ensure that the testsuite summary is not > unduly colorized; and since the summary at this point is still looks > like: > > ==================================================== > All 2 tests behaved as expected (1 expected failure) > ==================================================== > > those grepping checks have to stay IMNSHO. Okeydokey for now. Except of course that I suggested to not have '=' signs in the new output format. ;-) Cheers, Ralf
