Heres my thoughts:
* High level:
** I like the idea of letting cringe-worthy `try: ` obsolete out as a
bad idea going forward
** I like the idea of annotations of affects in moz.build files.
I want to avoid "AFFECTS" doing things like having 30 cases of
"windows" here. (e.g. win10, win64, windows10, 'win 10', etc) .
We should still (imo) support (until better exists) a command line
syntax for try, maybe `try: -v2` or something? This would require
some thought anyway.
We will need a way to normalize the list of `job:` and such as it
relates to AFFECTS* in moz.build to avoid adding bogus ones, and to
facilitate in realizing when ones already exist when adding new
things.
A way to do AND and OR in some way on the taskcluster `when` clause
could also help, e.g. if we want a set of jobs:docs but only when
taskcluster/**.rst is touched, or a python integration test run on
only windows, when the only file touched is a mozbase file only ever
used on windows, we won't need to run linux/osx mozbase tests.
To summarize, I'm a fan of any endeavor to make this more intuitive
and easier. And I'm not strongly against any color for the shed,
especially if someone other than me is painting it.
~Justin Wood (Callek)
On Fri, May 26, 2017 at 2:59 PM, Dustin Mitchell <dmitch...@mozilla.com> wrote:
> Greg sent some previous threads on this topic along, and I've been in
> some other related conversations. The topic of optimization overlaps
> substantially with try, so I'll mix that in too.
>
> There seems to be a reluctance to talk about this face-to-face, but
> there's some urgency in that we're spending a *lot* of cash right now
> doing unnecessary builds, since the current in-tree implementation is
> worse than the ad-hoc thing we had in Buildbot. So I'll outline a
> proposal here and we can all just agree on it! ;)
>
> For try, we will want three variants:
> - "there is no try, only do" -- just run the tasks appropriate for
> the push (machines figure out what to do)
> - "try this" -- an explicit list of desired task labels; this would
> be supported by a trychooser-like UI to generate the list, but the
> trivial case of "please run this exact task" would be easy to type
> directly. No surprises. Selected tasks will never be optimized away.
> - "trying my patience" -- the legacy try syntax, which seems to
> rankle everyone I talk to about it
>
> As far as optimization:
>
> We support two kinds of optimization: replacement, where we find a
> completed task that produced outputs we can re-use; and skipping,
> where we decide a task need not be performed at all. Future plans for
> replacement are clear, so we will limit consideration to skipping
> optimization. We'll further ignore SETA: it just causes tasks to be
> skipped except every seventh run, and doesn't apply on try.
>
> So what we're left with is a list of files that have been changed in
> this push, and a pile of tasks. The proposal is this:
>
> At the "job description" level, each task specifies when it should be
> executed as a disjunctive list - if any match, the task is executed.
> That can either be by specifying specific file patterns, or by
> specifying an "affected" key/value. Something like
>
> when:
> - files-changed: ['foo/bar/**/*.js']
> - change-affects: ['platform:windows', 'platform:macosx']
> ..or..
> when:
> - files-changed: ['**/*.rst']
> - change-affects: ['job:docs']
>
> We then add a specification of what "affects" mean to the moz.build files:
>
> with Files('**/*.rst'):
> AFFECTS += ['job:docs']
> with Files('**/*.js'):
> AFFECTS += ['job:eslint']
>
> the tricky bit is platforms[*], where everything-except-these affects
> a particular platform. As greg has said, we want to avoid manually
> decorating common files like `build/**` with every possible platform,
> as that violates the loose coupling between those components.
> Instead, let's let computers do that for us, using AFFECTS_ONLY:
>
> with Files('mobile/android/**'):
> AFFECTS_ONLY += ['platform:android']
> with Files('browser/**'):
> AFFECTS_ONLY += ['platform:desktop']
> with Files('stylo/**'): # or, per bholley, more specific patterns!
> AFFECTS_ONLY += ['platform:stylo']
>
> then do some post-processing to translate that so that *all* files
> affect platform:android except those which have AFFECTS_ONLY for some
> other platform. So `stylo/foo` would have AFFECTS set to
> ['platform:stylo'], but `build/foo` would have ['platform:android',
> 'platform:desktop', 'platform:stylo'].
>
> The details of implementing this efficiently are just that -- details.
>
> Note that this gets us the necessary condition that every file affects
> *something*, avoiding the case where a push to a particular file might
> trigger no tasks.
>
> So let's bikeshed on this - both the try division and the "affects"
> stuff - a little, and then we can re-evaluate the level of agreement
> and see if we should sit down in SFO.
>
> Dustin
>
> [*] This is the 72nd meaning of the term "platform" - I'm open to better ideas
> _______________________________________________
> dev-builds mailing list
> dev-builds@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-builds
_______________________________________________
dev-builds mailing list
dev-builds@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-builds
