On Mon, 30 Jul 2018 at 07:57:35 +0100, Rebecca N. Palmer wrote:
> else:
> errorcode |= 4
> - else:
> + elif 'trivial' not in t.restrictions:
> any_passed = True
>
> if 'breaks-testbed' in t.restrictions:
That's pretty much the implementation I had in mind. It might be clearer
to rename any_passed to non_trivial_passed or something.
> > > Do you[Simon McVittie] propose to add this[trivial] to autodep8-generated
> > > "check that we can
> > > import the top-level module" tests?
I hadn't actually thought of that, but yes, those tests should probably
be marked as trivial.
On Tue, 31 Jul 2018 at 22:56:03 +0100, Rebecca N. Palmer wrote:
> On 31/07/18 14:59, Paul Gevers wrote:
> > I mean, doing this
> > may already be enough to be used for gating.
>
> They would be used in that failing one would count as a fail (i.e. 15 days
> migration delay); the difference is that passing trivial tests counts as
> equivalent to having no tests (i.e. 5 days not 2).
Yes, this is what I had in mind for trivial tests:
If a package has real test coverage (e.g. dbus), and its tests pass,
then it seems safe to accelerate its migration; conversely, if its
tests fail, then something is wrong and we want to slow it down or block
it altogether.
If a package has only trivial tests (e.g. gdk-pixbuf), and they pass,
then they are not sufficient evidence to say the package actually works,
so we should wait a bit for users to report bugs, the same as if it
had no tests at all; but if trivial tests fail, they *are* sufficient
evidence to say the package *doesn't* work (assuming they are not also
marked flaky), so we should slow it down or block it.
(This has the consequence that a test that is marked both trivial and
flaky has no effect on gating at all, on the basis that it's sufficiently
trivial that passing is not a good indication that the package works,
but also sufficiently flaky that failing is not a good indication that
the package doesn't work. I think that's fine, and I don't think such
tests should be forbidden.)
smcv
