I would like to see failure rates expressed as a ratio of failures to test runs, but I recognise that this data may not be readily available and getting it may not be that important if we have a rough idea. These are a means for setting priorities, and so a rank works well.
If we have 100 tests, each with an expected failure rate of 3%, then the chance of a clear run is less than 5%. 3% is not an acceptable failure rate for a single test IMO. It does get harder to reproduce as the failure rate reduces, but logging can be added and inspected on the next failure, which will not be far away. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
