Re: Optimizing what runs on which push

2017-05-31 Thread Benjamin Smedberg
On Wed, May 31, 2017 at 11:08 AM, Dustin Mitchell wrote: > > > So the proposal addresses three issues: > - frustrating and ineffective try user interface > - high load and consequent long wait times > - elevated backout rate > Let me try and repeat back the assumptions/assertions I've heard,

Re: Optimizing what runs on which push

2017-05-31 Thread Andrew Halberstadt
On Wed, May 31, 2017 at 1:31 PM, Gregory Szorc wrote: > >> There is a facility in the tree for tagging tests (at least some suites) >> and running only tests with certain tags. However, that doesn't help on try >> very much because you can't tell try to run only a certain set of tags. >> > > Not

Re: Optimizing what runs on which push

2017-05-31 Thread Gregory Szorc
On Wed, May 31, 2017 at 6:26 AM, Benjamin Smedberg wrote: > I don't know if I'm the typical use-case, but the big problem for me is > that when I change something such as plugins, the jobs as currently > bucketed don't help much. There are reftests, crashtests, mochitest-plain, > and mochitest-br

Re: Optimizing what runs on which push

2017-05-31 Thread Dustin Mitchell
2017-05-31 9:26 GMT-04:00 Benjamin Smedberg : > Your ultimate goal is to save $$, and my ultimate goal is to reduce the time > to results to make the developer cycle faster (let's measure round-trip > times in minutes instead of hours). I don't believe that running subsets of > the current jobs wil

Re: Optimizing what runs on which push

2017-05-31 Thread Benjamin Smedberg
I don't know if I'm the typical use-case, but the big problem for me is that when I change something such as plugins, the jobs as currently bucketed don't help much. There are reftests, crashtests, mochitest-plain, and mochitest-browser-chrome which test plugin code paths, and the splitting means t

Re: Optimizing what runs on which push

2017-05-31 Thread Dustin Mitchell
I think this topic is big enough already without broadening it into "how can we make automation better". But getting some data from the survey sounds great! Maybe it makes sense to get down to the core question we have here: When you push to try, how often do you want: * to run every job relevan