I'm very interested in the try aspect of this (please CC me on relevant bugs).
I believe there is a relatively simple way taskcluster can wash its hand of try syntax completely and instead just use a list of tasks (or other similar data structure). We could still provide a legacy try syntax to people, but the syntax parser would be on the client side (e.g in a mach_command). If taskcluster does this, we can build all sorts of crazy trychooser thingamajigs. See this blog post for specific implementation steps as well as what a "fuzzyfinder" trychooser might look like: https://ahal.ca/blog/2017/fuzzy-try-chooser/ I agree with your 3 variants for try, I just don't think they should be implemented under /taskcluster. On Fri, May 26, 2017 at 2:59 PM, Dustin Mitchell <dmitch...@mozilla.com> wrote: > Greg sent some previous threads on this topic along, and I've been in > some other related conversations. The topic of optimization overlaps > substantially with try, so I'll mix that in too. > > There seems to be a reluctance to talk about this face-to-face, but > there's some urgency in that we're spending a *lot* of cash right now > doing unnecessary builds, since the current in-tree implementation is > worse than the ad-hoc thing we had in Buildbot. So I'll outline a > proposal here and we can all just agree on it! ;) > > For try, we will want three variants: > - "there is no try, only do" -- just run the tasks appropriate for > the push (machines figure out what to do) > - "try this" -- an explicit list of desired task labels; this would > be supported by a trychooser-like UI to generate the list, but the > trivial case of "please run this exact task" would be easy to type > directly. No surprises. Selected tasks will never be optimized away. > - "trying my patience" -- the legacy try syntax, which seems to > rankle everyone I talk to about it > > As far as optimization: > > We support two kinds of optimization: replacement, where we find a > completed task that produced outputs we can re-use; and skipping, > where we decide a task need not be performed at all. Future plans for > replacement are clear, so we will limit consideration to skipping > optimization. We'll further ignore SETA: it just causes tasks to be > skipped except every seventh run, and doesn't apply on try. > > So what we're left with is a list of files that have been changed in > this push, and a pile of tasks. The proposal is this: > > At the "job description" level, each task specifies when it should be > executed as a disjunctive list - if any match, the task is executed. > That can either be by specifying specific file patterns, or by > specifying an "affected" key/value. Something like > > when: > - files-changed: ['foo/bar/**/*.js'] > - change-affects: ['platform:windows', 'platform:macosx'] > ..or.. > when: > - files-changed: ['**/*.rst'] > - change-affects: ['job:docs'] > > We then add a specification of what "affects" mean to the moz.build files: > > with Files('**/*.rst'): > AFFECTS += ['job:docs'] > with Files('**/*.js'): > AFFECTS += ['job:eslint'] > > the tricky bit is platforms[*], where everything-except-these affects > a particular platform. As greg has said, we want to avoid manually > decorating common files like `build/**` with every possible platform, > as that violates the loose coupling between those components. > Instead, let's let computers do that for us, using AFFECTS_ONLY: > > with Files('mobile/android/**'): > AFFECTS_ONLY += ['platform:android'] > with Files('browser/**'): > AFFECTS_ONLY += ['platform:desktop'] > with Files('stylo/**'): # or, per bholley, more specific patterns! > AFFECTS_ONLY += ['platform:stylo'] > > then do some post-processing to translate that so that *all* files > affect platform:android except those which have AFFECTS_ONLY for some > other platform. So `stylo/foo` would have AFFECTS set to > ['platform:stylo'], but `build/foo` would have ['platform:android', > 'platform:desktop', 'platform:stylo']. > > The details of implementing this efficiently are just that -- details. > > Note that this gets us the necessary condition that every file affects > *something*, avoiding the case where a push to a particular file might > trigger no tasks. > > So let's bikeshed on this - both the try division and the "affects" > stuff - a little, and then we can re-evaluate the level of agreement > and see if we should sit down in SFO. > > Dustin > > [*] This is the 72nd meaning of the term "platform" - I'm open to better > ideas > _______________________________________________ > dev-builds mailing list > dev-builds@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-builds >
_______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
