On Thursday 23 June 2011, Stefano Lattarini wrote: > On Thursday 23 June 2011, Ralf Wildenhues wrote: > > * Stefano Lattarini wrote on Thu, Jun 23, 2011 at 09:52:55AM CEST: > > > > > > [SNIP] > > > > > > the authors of third-party test drivers *must* allow the direct writing > > > of `:test-result:' directives in the driver output -- this is the only > > > way to declare test results that are understood and accounted for by the > > > new Automake test harness. > > > > So, we are doing inband signaling. > > > Yes. > > > But Automake's own testsuite is not a TAP or a subunit one. Thus it > > does not need to do inband-signaling. Right? > > > Strictly speaking, it doesn't have to. But the point of my patches so > far is exactly to provide a minimal in-band signaling support that can > be used by *all* test drivers that implements our published APIs -- so > that the logs produced by those drivers can be parsed by the generic > 'parallel-tests' harness when creating the final testsuite report. > Elaborating a little more on this, we might decide that the declarations of the test results can be placed only on the top of the log files (well, almost -- the must go *after* the first line in the log, line which *must* contain the declaration of the "global" test script outcome). The parsing of these "result-declaring" directives will then stop at the first line that is not a recognized directive. This would retain full backward compatibility, at the expense of a little more work on the part of the test drivers (but *not* on the part of our default 'test-driver' script, which already does something like that anyway). I had thought of proposing something like this in follow-up patches, but since such a change will make the testsuite patch we are discussing here moot, we can implement it right away instead, and save ourselves some trouble later. WDYT?
> > The reason I'm harping on this is that, if all existing users of > > Automake test drivers are updated to use the same kind of inband > > signaling, then they all are liable to similar bad signals. That > > is a potential source for regression. > > This wouldn't be anymore the case with my proposal above. > Correct; that's possible, even if very unlikely. BTW, I had proposed > to use of `:am-test-result:' instead of `:test-results:' results exactly > to make such "spurious" signals less likely (but they'd be still possible > of course, so that's not a great solution). > > Another possibility would be to make the RST field used to declare > in-band test results configurable (at either Automake or make time); > so that the users could protect themselves from spurious test results > by defining that field to a value they know will never be emitted in > the stdout/stderr of their tests. > And this also is made moot by my proposal above. > > [SNIP] > Regards, Stefano
