Le 15/02/2019 à 18:29, Xavier a écrit : > Le 15/02/2019 à 17:57, gregor herrmann a écrit : >> On Wed, 13 Feb 2019 21:23:00 +0100, Xavier Guimard wrote: >> >>> Some suggestions for pkg-js-autopkgtest based on pkg-js-autopkgtest >>> discussion with autodep8 maintainers: >>> - tests skipped should return a 77 exit code and all tests marked as >>> "Restrictions: skippable". It avoids to consider that a test succeeds >>> if maintainer skipped it, but needs a merge request to autodep8. See >>> >>> https://salsa.debian.org/ci-team/autodep8/blob/master/support/nodejs/generate >>> (changed by MR !11) >>> - runtime-deps* tests should be tagged as "Restrictions: superficial" >>> since these tests don't really test package features but just Perl >>> syntax >>> >>> Then with this 2 changes, if "build-deps.d" is skipped, success won't >>> give the benefit of 3-days-reduce. >> >> Thanks for your work and the pull request [0]! >> >> Some thoughts and more questions: >> >> For the skippable part: >> - If I understand this correctly (from your text above and the spec >> [1]) then a skipped syntax.t and use.t would also lead to losing >> the benefit of faster migration? Do we want this? > > The benefit will be lost only if smoke test is skipped. I think it's a > good thing (other tests are "superficial" <=> no benefit). Today if this > test is skipped, it is considered by autopkgtest as "success" > >> Or does it just have no influence? > > skippable has no influence if test succeed, but don't consider that test > has failed if a 77 code is returned > >> - As for the implementation in [0]: >> not sure if the "exit 0" in smoke is correct >> - What about the skipped tests within use.t and syntax.t? Should they >> or some of them also exit 77? > > runner do it for them. I didn't modify them if all is skipped as it has > no effect on a test marked as "superficial": 0 or 77 gives the same > result: no benefit, no penalty > >> For the superficial part: >> Hm, yeah, use.t and syntax.t don't test that everything in the >> package is fully functional; still, this "superficial" feels a bit >> weird. But probably it's correct according to [1]. >> >> In general I still don't have the full picture of what benefits and >> penalties for testing migration will result from which combination of >> the changes under which circumstances. > > The only effect of this is that if smoke test is skipped, there is no > benefit. And I think it's more clear to have the real result:
I found the exact message, it is very clear: # EXAMPLE 1, SKIP use.t => benefit OK autopkgtest [18:23:17]: @@@@@@@@@@@@@@@@@@@@ summary command1: PASS command2: SKIP exit status 77 and marked as skippable command3: PASS (superficial) # EXAMPLE 2, SKIP smoke => no benefit command1: SKIP exit status 77 and marked as skippable command2: PASS (superficial) command3: PASS (superficial) # EXAMPLE 3, real failure in smoke => penalty command1: FAIL command2: PASS (superficial) command3: PASS (superficial) # EXAMPLE 4, real failure in use.t => penalty command1: PASS command2: FAIL command3: PASS (superficial) Cheers, Xavier
