On 4/9/14, 2:07 PM, Karl Tomlinson wrote:
Gregory Szorc writes:
2) Run marked intermittent tests multiple times. If it works all
25 times, fail the test run for inconsistent metadata.
We need to consider intermittently failing tests as failed, and we
need to only test things that always pass.
I strongly reject the second part of that sentence. That world leads to
a state where we know less and don't test as much. It throws away a huge
amount of valuable data.
We can't rely on statistics to tell us about changes in test
results without huge expense. If we have many intermittent tests
running many times, then sometimes they will fail in all 25 runs
and sometimes they will pass in all 25 runs.
I threw out 25 as a placeholder. We could have metadata differentiate
between tests that are expected to fail 1% or 50% and treat them
appropriately.
If you are concerned about costs of testing, I submit to you Microsoft's
costs for not adequately testing the browser choice screen: a €561
million fine [1]. I don't want to be the person explaining to the
lawyers that we disabled a test that was verifying our legal compliance
with a law or contract because it failed intermittently 2% of the time.
[1] http://europa.eu/rapid/press-release_IP-13-196_en.htm
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
