On 03/08/15 23:41, Josh Matthews wrote:
Hello everyone! Let's talk about when and how I would like us to use
"@bors-servo: retry" in our PRs.
* Which tests actually failed? Is each one filed in servo/servo as a
known intermittent failure?
* For the tests which are not previously known intermittents, what is
their output on this run? Are there any errors or panics in the log that
look related to the PR's changes?
* Is there anything in the test itself that looks like the PR's changes
could affect it? Pay particular attention to tests that report an
unexpected pass.
* Have issues been filed for each test which intermittently failed for
reasons that cannot be ascribed to this PR's changes? Are these issues
in the form "Intermittent [timeout|fail|crash|pass] for
/path/to/test.html" with the I-intermittent label applied and the
complete output of that test from the failing run copied and pasted?
Build logs disappear; github comments are forever.
* Has justification been provided in this "@bors-servo: retry" comment
for why a retry is necessary. Preferably this happens in the form of "*
#[issue number 1], #[issue number 2]".
A lot of these issues look like "test results are hard to find". In the
spirit of automation and tooling, it might be worth investing some time
in making this easier. I can think of a couple of possibilities:
1) Write a bot that reads the structured log output and formats any
unexpected results as a github comment (much like jdm-bot already does).
For bonus point search for existing I-intermittents matching the test
and result, and link that issue in the comment. For bonus bonus points
instruct homu to do a single retry automatically if all unexpected
results match known issues.
2) Start posting the results of test runs to treeherder, which will
surface the failures in a clearer way than the buildbot waterfall and
will hopefully in the near future get a measure of auto-matching test
results against known intermittents (this will probably be internal
rather than using any bugtracker as a backend, so it should be possible
to also link specific intermittents to github issues if you want that
feature).
_______________________________________________
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo
