I think this topic is big enough already without broadening it into "how can we make automation better". But getting some data from the survey sounds great! Maybe it makes sense to get down to the core question we have here:
When you push to try, how often do you want: * to run every job relevant to the changes you have made [ ] never [ ] rarely [ ] sometimes [ ] often [ ] always] * to run a specific job or set of jobs [ ] never [ ] rarely [ ] sometimes [ ] often [ ] always] * to run all jobs for one or more platforms [ ] never [ ] rarely [ ] sometimes [ ] often [ ] always] Or something like that? 2017-05-30 21:21 GMT-04:00 Mike Hommey <m...@glandium.org>: > On Tue, May 30, 2017 at 05:25:20PM -0700, Gregory Szorc wrote: >> On Thu, May 11, 2017 at 10:05 AM, <dmitch...@mozilla.com> wrote: >> >> > Background: >> > https://bugzilla.mozilla.org/show_bug.cgi?id=1359942 >> > >> > As jobs move to taskcluster, we have an improved opportunity to do some >> > smarter scheduling of what jobs to run on what sort of push. Of course, >> > it's a thorny subject: optimizing away a task that should run may let a bad >> > push show green, while a subsequent push bears responsibility for the >> > orange it introduces. >> > >> > One of the more common expectations is that pushes that only change a >> > directory affecting one platform should not cause other platforms' tasks to >> > run. >> > >> > In the bug above, I have proposed a method of identifying pushes >> > "affecting" a particular platform, and Greg has raised some concerns about >> > the generality of my solution. I'm happy to generalize, but I would like >> > to keep the process in motion rather than let the perfect be the enemy of >> > the good. >> > >> > To that end, I'd like some further feedback on implementing this sort of >> > optimization support. >> > >> > If there's sufficient interest, then this is probably something we could >> > set up a time to talk about in SFO in June. >> > >> >> I still owe a proper reply to everything in this thread. But as I'm >> preparing to send out another Firefox developer survey, I'm looking at the >> old one we conducted and there are some results that seemingly justify >> doing work to intelligently run things based on what changed. >> >> One of the questions on the last survey was "Thinking of running automated >> tests, rank the following potential improvements in terms of their impact >> on your productivity." "Determine and run relevant tests based on what >> source files have been modified" was one of the most wanted improvements - >> right up there with "make try runs really fast so I can effectively iterate >> on automated tests using try instead." > > FWIW, I recently added a unit test for Firefox. On try, I essentially > had to run the whole corresponding test suite (browser-chrome), instead > of just the block that contains the test, because it's almost impossible > to figure out which one it's going to run in. > > Making /that/ less painful would go a long way. > > Mike _______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
