Docs might be useful, but they won't cover all cases. Sometimes you get certain failures that only show up for a short time due to infra weirdness. I often find myself asking on IRC if a particular kind of failure is known, and often the answer is a variation on "yes, that's just been happening recently".
Nick On Tue, Jul 12, 2016 at 6:14 AM, Jonathan Griffin <jgrif...@mozilla.com> wrote: > I think what you're asking for is a description of what sheriffs do when > reviewing a push, so you can do it yourself for your own try push. Sheriffs, > do we have any such documentation? > > On Mon, Jul 11, 2016 at 11:30 AM, allassopraise . <allassopra...@gmail.com> > wrote: >> >> The suggestions for improving the way treeherder reports are good. >> >> Nevertheless, for now, I still would like some documentation or >> pointers on how to interpret what is there, but most especially how to >> know whether an error is due to the changeset that's been pushed. >> >> *Somebody* is obviously able to do it, as I can tell when a patch I've >> written gets pushed, then backed out due to results on treeherder. >> >> On 7/11/16, Jonathan Griffin <jgrif...@mozilla.com> wrote: >> > Yes, a clear Yes/No answer for Try pushes is something we'd like to be >> > able >> > to report. >> > >> > Currently, interpreting Try results can be hard for a few reasons: >> > >> > 1 - intermittent test failures make it hard to separate failures your >> > push >> > has caused vs pre-existing failures >> > 2 - people sometimes push to try from bad heads (like a mozilla-inbound >> > push that will eventually get backed out) >> > 3 - it's not always clear whether you should care about try failures of >> > Tier 2 jobs >> > >> > For 1, as you noted, we have started implementing automatic >> > classification >> > in Treeherder. We're expanding this to cover more types of failures; see >> > bug 1241940. The goal is to have it able to identify upwards of 80% of >> > known failures automatically, which should make interpreting try results >> > considerably easier. >> > >> > For 2, making sure you push to try only from the current tip of m-c will >> > avoid this problem, or using MozReview's autoland to try, which will >> > automatically do the same thing. >> > >> > For 3, I think we should distinguish tiers better in Treeherder, and >> > include some info about tolerance for failures in the UI. >> > >> > Jonathan >> > >> > On Mon, Jul 11, 2016 at 4:18 AM, Nicolas B. Pierron < >> > nicolas.b.pier...@mozilla.com> wrote: >> > >> >> On 07/10/2016 06:48 PM, allassopraise . wrote: >> >> >> >>> For example, I believe the orange and red links indicate an error, but >> >>> how do I know if they pertain to my push or not? >> >>> >> >> >> >> I have the same issue, and in many cases I feel clueless about the >> >> oranges >> >> & reds. At the end I end-up pushing to inbound, even if I feel that I >> >> had >> >> a higher number of oranges / reds on try than I would expect to see on >> >> inbound. >> >> >> >> I see that we already have a classifier which automatically mark some >> >> of >> >> the intermittent, and if this tool were able to classify the remaining >> >> issue, this would be quite helpful. >> >> >> >> I guess, what I am missing with Try pushes, is a clear Yes/No answer. >> >> >> >> -- >> >> Nicolas B. Pierron >> >> >> >> >> >> _______________________________________________ >> >> dev-builds mailing list >> >> dev-builds@lists.mozilla.org >> >> https://lists.mozilla.org/listinfo/dev-builds >> >> >> > > > > > _______________________________________________ > dev-builds mailing list > dev-builds@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-builds > _______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
