On Fri, 15 Feb 2019 18:29:14 +0100, Xavier wrote: > > For the skippable part: > > - If I understand this correctly (from your text above and the spec > > [1]) then a skipped syntax.t and use.t would also lead to losing > > the benefit of faster migration? Do we want this? > The benefit will be lost only if smoke test is skipped. I think it's a > good thing (other tests are "superficial" <=> no benefit). Today if this > test is skipped, it is considered by autopkgtest as "success"
Ah, I see. Ok, a lost benefit only for the skipped smoke tests probably makes sense. > > - As for the implementation in [0]: > > not sure if the "exit 0" in smoke is correct This still confuses me. Shouldn't it "exit $?" or just nothing (line 174)? > > - What about the skipped tests within use.t and syntax.t? Should they > > or some of them also exit 77? > runner do it for them. Well, only partially. First of all, runners can run more than one test in a subdirectory (even if we currently only have 3 files in 4 subdirectories, with 3 times 1 and 1 time zero), second, there are several places in syntax.t and use.t were all or parts of the tests are skipped. -- But: > I didn't modify them if all is skipped as it has > no effect on a test marked as "superficial": 0 or 77 gives the same > result: no benefit, no penalty … this "no benefit, no penalty" makes it indeed kind of moot :) > > In general I still don't have the full picture of what benefits and > > penalties for testing migration will result from which combination of > > the changes under which circumstances. > > The only effect of this is that if smoke test is skipped, there is no > benefit. And I think it's more clear to have the real result: > > # EXAMPLE 1, SKIP use.t => benefit OK > autopkgtest [18:23:17]: @@@@@@@@@@@@@@@@@@@@ summary > command1: PASS > command2: FAIL exit 77 (marked as skippable) # I don't remember the > # exact message > command3: PASS (superficial) > > # EXAMPLE 2, SKIP smoke => no benefit > command1: FAIL exit 77 (marked as skippable) > command2: PASS (superficial) > command3: PASS (superficial) > > # EXAMPLE 3, real failure in smoke => penalty > command1: FAIL > command2: PASS (superficial) > command3: PASS (superficial) > > # EXAMPLE 4, real failure in use.t => penalty > command1: PASS > command2: FAIL > command3: PASS (superficial) Thanks alot for those examples, they make it indeed easier for me to understand the effects! So, hm, yeah, I guess that all makes sense … I hope someone else also has some minutes to think it through :) (Random note, so we don't forget it: There are a few adjusted copies of the autodep8 file in various packages as debian/tests/control which should also be adjusted, at least in git for the next upload.) Cheers, gregor -- .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe `- NP: Josh With: Milk cow blues
signature.asc
Description: Digital Signature
