Is there a mechanism in place to detect when disabled intermittent tests
have been fixed?
eg, every so often you could rerun disabled tests individually a bunch
of times. Or if you can distinguish which tests are failing, run them
all a bunch of times and pick apart the wreckage to see which ones are
now consistently passing. I'm not suggesting those, just using them as
example solutions to illustrate what I mean.
On 03/07/2017 10:33 AM, Honza Bambas wrote:
I presume that when a test is disabled a bug is filed and triaged
within the responsible team as any regular bug. Only that way we
don't forget and push on fixing it and returning back to the wheel.
Are there also some data or stats how often tests having a strong
orange factor catch actual regressions? I.e. fail a different way
than the filed "intermittent" one and uncover a real bug leading to a
patch back out or filing of a regular functionality regression bug.
If that number is found to be high(ish) for a test, the priority of
fixing it after its disabling should be raised.
-hb-
On 07/03/2017 17:10, jma...@mozilla.com wrote:
On Tuesday, March 7, 2017 at 10:37:12 AM UTC-5, Boris Zbarsky wrote:
On 3/7/17 9:26 AM, jma...@mozilla.com wrote:
We find that we are fixing 35% of the bugs and disabling 23% of them.
Is there a credible plan for reenabling the ones we disable?
-Boris
Great question, we do not have a credible plan, but we will have a
quick way to see what is disabled:
https://goo.gl/EfDKxY
I would like to build a dashboard when possible to outline per
component which test cases exist, which are disabled, and what
related intermittent bugs there are. Part of annotating moz.build
files with BUG_COMPONENTS is to make it easier to associate all test
cases with components whoch would help with a dashboard.
Do you have suggestions for how to ensure we keep up with the
disabled tests?
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
