Yes, a clear Yes/No answer for Try pushes is something we'd like to be able to report.
Currently, interpreting Try results can be hard for a few reasons: 1 - intermittent test failures make it hard to separate failures your push has caused vs pre-existing failures 2 - people sometimes push to try from bad heads (like a mozilla-inbound push that will eventually get backed out) 3 - it's not always clear whether you should care about try failures of Tier 2 jobs For 1, as you noted, we have started implementing automatic classification in Treeherder. We're expanding this to cover more types of failures; see bug 1241940. The goal is to have it able to identify upwards of 80% of known failures automatically, which should make interpreting try results considerably easier. For 2, making sure you push to try only from the current tip of m-c will avoid this problem, or using MozReview's autoland to try, which will automatically do the same thing. For 3, I think we should distinguish tiers better in Treeherder, and include some info about tolerance for failures in the UI. Jonathan On Mon, Jul 11, 2016 at 4:18 AM, Nicolas B. Pierron < nicolas.b.pier...@mozilla.com> wrote: > On 07/10/2016 06:48 PM, allassopraise . wrote: > >> For example, I believe the orange and red links indicate an error, but >> how do I know if they pertain to my push or not? >> > > I have the same issue, and in many cases I feel clueless about the oranges > & reds. At the end I end-up pushing to inbound, even if I feel that I had > a higher number of oranges / reds on try than I would expect to see on > inbound. > > I see that we already have a classifier which automatically mark some of > the intermittent, and if this tool were able to classify the remaining > issue, this would be quite helpful. > > I guess, what I am missing with Try pushes, is a clear Yes/No answer. > > -- > Nicolas B. Pierron > > > _______________________________________________ > dev-builds mailing list > dev-builds@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-builds >
_______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
