Very interesting, thank you! Would there be a way to add an environment variable or harness flag to run all tests in chaos mode?
On Thu, Jun 4, 2015 at 5:31 PM, Chris Peterson <[email protected]> wrote: > On 6/4/15 11:32 AM, [email protected] wrote: > >> I just landed bug 1164218 on inbound, which adds the ability to run >> individual mochitests and reftests in chaos mode. (For those unfamiliar >> with chaos mode, it's a feature added by roc a while back that makes >> already-random things more random; see [1] or bug 955888 for details). >> >> The idea with making it available per-test is that new tests should be >> written and tested locally/on try with chaos mode enabled, to flush out >> possible intermittent failures faster. Ideally we should also land them >> with chaos mode enabled. At this time we're still not certain if this will >> provide a lot of value (i.e. if chaos-mode-triggered failures are >> representative of real bugs) so it's not mandatory to make your tests run >> in chaos mode, but please do let me know if you try enabling it on your >> test and are either successful or not. We need to collect more data on the >> usefulness of this to see where we should take it. If it does turn out to >> be valuable, my hope is that we can start making pre-existing tests >> chaos-mode enabled as well, and eventually reduce the intermittent failure >> rate. >> > > Will chaos mode enabled tests run on Try and release branches? > > We don't know if chaos mode test failures are representative of real bugs, > but could chaos mode hide bugs that only reveal themselves when users run > without chaos mode? > > > > See [2] for an example of how to enable chaos mode in your tests. >> Basically you can add chaos-mode to the reftest.list file for reftests, or >> call SimpleTest.testInChaosMode() for mochitests. >> >> If you do run into intermittent failures, the best way to debug them is >> usually to grab a recording of the failure using rr [3] and then debug the >> recording to see what was going on. This only works on Linux (and has some >> hardware requirements as well) but it's a really great tool to have. >> >> Cheers, >> kats >> >> [1] http://robert.ocallahan.org/2014/03/introducing-chaos-mode.html >> [2] https://hg.mozilla.org/integration/mozilla-inbound/rev/89ac61464a45 >> [3] http://rr-project.org/ or https://github.com/mozilla/rr/ >> > > _______________________________________________ > dev-platform mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list [email protected] https://lists.mozilla.org/listinfo/dev-platform
