Also sprach Andrew Halberstadt <ahalberst...@mozilla.com>:
> I don't have any data to back this up, but my suspicion is that a
> large percentage of backouts had try runs, but said try runs
> didn't run the jobs that failed and caused the backout.
I ascribe to this explanation. This has happened to me a few times
because I have not run all the tests relevant to a change. It has
either been due to forgetfulness when constructing the try syntax,
or because Marionette has been picked up and used in some new,
inventive way in a component not covered by my default syntax.
> For example, you could either:
>
> A) Do a full try run then push, almost guaranteeing you won't be
> backed out. But then you run every job twice and take longer to
> complete your bug, a significant cost.
>
> B) Do a partial try run, running X% of the jobs yielding a Y%
> chance of being backed out.
A third option would be to introduce a new try syntax to have the CI
pick out the right test jobs to run based on the files changed.
Using mach it is possible to extract (known) test dependency
information about source files:
% ./mach file-info dep-tests testing/marionette/driver.js
testing/marionette/driver.js:
Test file patterns:
dom/network/tests/marionette/**
testing/marionette/**
dom/events/test/marionette/**
testing/marionette/harness/marionette_harness/tests/unit/**
dom/system/gonk/tests/marionette/**
If it was possible to determine what try syntax covers those
directories, it might be possible to run an intermediary pre-step on
TaskCluster to generate the appropriate try syntax.
It strikes me as an attractive feature if our CI could figure out
what the appropriate tests to run are based on the changes in the
push. I don’t know how well this would work or how accurate the
test dependency information from mach is, but it seems worth
experimenting with.
As an aside, I filed a bug to have `./mach test SOURECFILE` pick up
the known test dependencies a few weeks ago:
https://bugzilla.mozilla.org/show_bug.cgi?id=1345942
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
