On 16 February 2016 at 16:38, Nick Fitzgerald <nfitzger...@mozilla.com> wrote: > It seems like try/tbpl could automatically detect new test files and run > them N times. That way, the developer doesn't have to do it manually, so it > is less "intimidating" and also less likely to be skipped by accident or > forgotten.
We employed this strategy with some success at Opera. New tests would be put into quarantine and run N times so that their stability could be determined. Test stability was value from 0.0 to 1.0, where 1.0 indicated absolute stableness. A quarantined test would be run alongside other stable tests, but would be excluded from the regression report. The regression report was roughly the equivalent of a green try run indicating that it’s okay to check in your code to inbound; except it showed that your change caused no _additional_ test failures compared to before. Once the test had reached a certain threshold it would either be deemed stable, and included in regression reports, or unstable, and disabled permanently. As I understand it, one of the hurdles to implementing a similar system at Mozilla is that we do not have a concept of test uniqueness and a central database collecting tests and their results, but perhaps this can be retrospectively mitigated using ActiveData (https://wiki.mozilla.org/Auto-tools/Projects/ActiveData). _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
