On 3/4/13 5:09 PM, Dave Mandelin wrote:
We already don't back back out changes for regressing a benchmark like
we back them out for regressing tests. I think this is at least
partially because a general sentiment that not all of our benchmarks
correlate strongly to what they're trying to measure.
I know this has been a hot topic lately. I think getting more clarity on this
would be great, *if* of course we could have an answer that was both
operationally beneficial and clear, which seems to be incredibly difficult.
But this thread gives me a new idea. If each test run in automation had an
owner (as I suggested elsewhere), how about also making the owners responsible
for informing the sheriffs about what to do in case of regression? If the
owners know the test is reliable and measures something important, they can ask
for monitoring and presumptive backout. If not, they can ask sheriffs to ignore
the test, inform and coordinate with the owning team, inform the landing person
only, or some other action.
This should be annotated in the tests themselves, IMO. We could even
have said annotation influence the color on TBPL. A well-written test
harness could also re-run failing tests to see if failures are constant
or intermittent. We could also introduce "expectations" instead of
"assertions" and have soft/expectancy failures for things like assertion
count mismatch. IMO we should be focusing on lessening the burden on the
sheriffs and leaving them to focus on real problems. There's so much
more we can be doing with our test infrastructure...
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
