On Wednesday, September 07, 2011 05:28:15 PM Diego Novillo wrote:
> One of the most vexing aspects of GCC development is dealing with
> failures in the various testsuites. In general, we are unable to
> keep failures down to zero. We tolerate some failures and tell
> people to "compare your build against a clean build".
>
> This forces developers to either double their testing time by
> building the compiler twice or search in gcc-testresults and hope
> to find a relatively similar build to compare against.
>
> Additionally, the marking mechanisms in DejaGNU are generally
> cumbersome and hard to add. Even worse, depending on the
> controlling script, there may not be an XFAIL marker at all.
>
> So, while we would ideally keep NO failures in the testsuite, the
> reality is that we are content with having KNOWN failures. For a
> given set of failures out of 'make check', I would like to have a
> simple filtering mechanism that prunes the known failures out.
>
> Desired features:
>
> - List of known failures lives in SVN.
> - Each target can have its own list.
> - Supports ignoring FAIL, UNRESOLVED and XPASS results.
> - Supports pattern matching to glob sets of failures.
> - Co-exists with the existing XFAIL support in DejaGNU.
> - Supports flaky tests.
> - Supports timestamps to avoid having tests in a knonw-to-fail
> state forever.
>
> In terms of implementation, this filter could be part of 'make
> check'. We'd pipe make check's output to it and it would decide
> whether to emit FAIL/UNRESOLVED/XPASS lines based on the black
> list.
>
> I could also make this a post-check filter that runs on all the
> generated <tool>.sum files. The filter could live in
> <src>/contrib and be used on demand.
>
> I am not thrilled about the prospect of implementing this in
> DejaGNU directly.
>
> Thoughts?
good idea.
I have my own homegrown script that builds gcc and outputs something like:
trunk: successfull (New passes: 32, new fails: 0, time: 1:55 h)
4.6: successfull (New passes: 28, new fails: 12, time: 1:43 h)
I'll send it to you offline,
Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
