The suggestions for improving the way treeherder reports are good. Nevertheless, for now, I still would like some documentation or pointers on how to interpret what is there, but most especially how to know whether an error is due to the changeset that's been pushed.
*Somebody* is obviously able to do it, as I can tell when a patch I've written gets pushed, then backed out due to results on treeherder. On 7/11/16, Jonathan Griffin <jgrif...@mozilla.com> wrote: > Yes, a clear Yes/No answer for Try pushes is something we'd like to be able > to report. > > Currently, interpreting Try results can be hard for a few reasons: > > 1 - intermittent test failures make it hard to separate failures your push > has caused vs pre-existing failures > 2 - people sometimes push to try from bad heads (like a mozilla-inbound > push that will eventually get backed out) > 3 - it's not always clear whether you should care about try failures of > Tier 2 jobs > > For 1, as you noted, we have started implementing automatic classification > in Treeherder. We're expanding this to cover more types of failures; see > bug 1241940. The goal is to have it able to identify upwards of 80% of > known failures automatically, which should make interpreting try results > considerably easier. > > For 2, making sure you push to try only from the current tip of m-c will > avoid this problem, or using MozReview's autoland to try, which will > automatically do the same thing. > > For 3, I think we should distinguish tiers better in Treeherder, and > include some info about tolerance for failures in the UI. > > Jonathan > > On Mon, Jul 11, 2016 at 4:18 AM, Nicolas B. Pierron < > nicolas.b.pier...@mozilla.com> wrote: > >> On 07/10/2016 06:48 PM, allassopraise . wrote: >> >>> For example, I believe the orange and red links indicate an error, but >>> how do I know if they pertain to my push or not? >>> >> >> I have the same issue, and in many cases I feel clueless about the >> oranges >> & reds. At the end I end-up pushing to inbound, even if I feel that I >> had >> a higher number of oranges / reds on try than I would expect to see on >> inbound. >> >> I see that we already have a classifier which automatically mark some of >> the intermittent, and if this tool were able to classify the remaining >> issue, this would be quite helpful. >> >> I guess, what I am missing with Try pushes, is a clear Yes/No answer. >> >> -- >> Nicolas B. Pierron >> >> >> _______________________________________________ >> dev-builds mailing list >> dev-builds@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-builds >> > _______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
