I don't know if I'm the typical use-case, but the big problem for me is that when I change something such as plugins, the jobs as currently bucketed don't help much. There are reftests, crashtests, mochitest-plain, and mochitest-browser-chrome which test plugin code paths, and the splitting means that I pretty much need to run all of those suites on try to get adequate coverage.
There is a facility in the tree for tagging tests (at least some suites) and running only tests with certain tags. However, that doesn't help on try very much because you can't tell try to run only a certain set of tags. Your ultimate goal is to save $$, and my ultimate goal is to reduce the time to results to make the developer cycle faster (let's measure round-trip times in minutes instead of hours). I don't believe that running subsets of the current jobs will solve either of those goals. Maybe it's a step along the path, but I don't see how that fits together yet. Related to this, you need to remember one of the primary functions of try is to validate changes before landing on inbound/autoland, and so reduce the backout rate from sheriffs. Running subsets of tests will increase the backout rate. I think that's probably ok, but we need to be aware of this social/workflow impact as it's not just a technical decision. --BDS On Wed, May 31, 2017 at 8:47 AM, Dustin Mitchell <dmitch...@mozilla.com> wrote: > I think this topic is big enough already without broadening it into > "how can we make automation better". But getting some data from the > survey sounds great! Maybe it makes sense to get down to the core > question we have here: > > When you push to try, how often do you want: > * to run every job relevant to the changes you have made > [ ] never [ ] rarely [ ] sometimes [ ] often [ ] always] > * to run a specific job or set of jobs > [ ] never [ ] rarely [ ] sometimes [ ] often [ ] always] > * to run all jobs for one or more platforms > [ ] never [ ] rarely [ ] sometimes [ ] often [ ] always] > > Or something like that? > > 2017-05-30 21:21 GMT-04:00 Mike Hommey <m...@glandium.org>: > > On Tue, May 30, 2017 at 05:25:20PM -0700, Gregory Szorc wrote: > >> On Thu, May 11, 2017 at 10:05 AM, <dmitch...@mozilla.com> wrote: > >> > >> > Background: > >> > https://bugzilla.mozilla.org/show_bug.cgi?id=1359942 > >> > > >> > As jobs move to taskcluster, we have an improved opportunity to do > some > >> > smarter scheduling of what jobs to run on what sort of push. Of > course, > >> > it's a thorny subject: optimizing away a task that should run may let > a bad > >> > push show green, while a subsequent push bears responsibility for the > >> > orange it introduces. > >> > > >> > One of the more common expectations is that pushes that only change a > >> > directory affecting one platform should not cause other platforms' > tasks to > >> > run. > >> > > >> > In the bug above, I have proposed a method of identifying pushes > >> > "affecting" a particular platform, and Greg has raised some concerns > about > >> > the generality of my solution. I'm happy to generalize, but I would > like > >> > to keep the process in motion rather than let the perfect be the > enemy of > >> > the good. > >> > > >> > To that end, I'd like some further feedback on implementing this sort > of > >> > optimization support. > >> > > >> > If there's sufficient interest, then this is probably something we > could > >> > set up a time to talk about in SFO in June. > >> > > >> > >> I still owe a proper reply to everything in this thread. But as I'm > >> preparing to send out another Firefox developer survey, I'm looking at > the > >> old one we conducted and there are some results that seemingly justify > >> doing work to intelligently run things based on what changed. > >> > >> One of the questions on the last survey was "Thinking of running > automated > >> tests, rank the following potential improvements in terms of their > impact > >> on your productivity." "Determine and run relevant tests based on what > >> source files have been modified" was one of the most wanted > improvements - > >> right up there with "make try runs really fast so I can effectively > iterate > >> on automated tests using try instead." > > > > FWIW, I recently added a unit test for Firefox. On try, I essentially > > had to run the whole corresponding test suite (browser-chrome), instead > > of just the block that contains the test, because it's almost impossible > > to figure out which one it's going to run in. > > > > Making /that/ less painful would go a long way. > > > > Mike > _______________________________________________ > dev-builds mailing list > dev-builds@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-builds >
_______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
