Greg sent some previous threads on this topic along, and I've been in
some other related conversations. The topic of optimization overlaps
substantially with try, so I'll mix that in too.
There seems to be a reluctance to talk about this face-to-face, but
there's some urgency in that we're spending a *lot* of cash right now
doing unnecessary builds, since the current in-tree implementation is
worse than the ad-hoc thing we had in Buildbot. So I'll outline a
proposal here and we can all just agree on it! ;)
For try, we will want three variants:
- "there is no try, only do" -- just run the tasks appropriate for
the push (machines figure out what to do)
- "try this" -- an explicit list of desired task labels; this would
be supported by a trychooser-like UI to generate the list, but the
trivial case of "please run this exact task" would be easy to type
directly. No surprises. Selected tasks will never be optimized away.
- "trying my patience" -- the legacy try syntax, which seems to
rankle everyone I talk to about it
As far as optimization:
We support two kinds of optimization: replacement, where we find a
completed task that produced outputs we can re-use; and skipping,
where we decide a task need not be performed at all. Future plans for
replacement are clear, so we will limit consideration to skipping
optimization. We'll further ignore SETA: it just causes tasks to be
skipped except every seventh run, and doesn't apply on try.
So what we're left with is a list of files that have been changed in
this push, and a pile of tasks. The proposal is this:
At the "job description" level, each task specifies when it should be
executed as a disjunctive list - if any match, the task is executed.
That can either be by specifying specific file patterns, or by
specifying an "affected" key/value. Something like
when:
- files-changed: ['foo/bar/**/*.js']
- change-affects: ['platform:windows', 'platform:macosx']
..or..
when:
- files-changed: ['**/*.rst']
- change-affects: ['job:docs']
We then add a specification of what "affects" mean to the moz.build files:
with Files('**/*.rst'):
AFFECTS += ['job:docs']
with Files('**/*.js'):
AFFECTS += ['job:eslint']
the tricky bit is platforms[*], where everything-except-these affects
a particular platform. As greg has said, we want to avoid manually
decorating common files like `build/**` with every possible platform,
as that violates the loose coupling between those components.
Instead, let's let computers do that for us, using AFFECTS_ONLY:
with Files('mobile/android/**'):
AFFECTS_ONLY += ['platform:android']
with Files('browser/**'):
AFFECTS_ONLY += ['platform:desktop']
with Files('stylo/**'): # or, per bholley, more specific patterns!
AFFECTS_ONLY += ['platform:stylo']
then do some post-processing to translate that so that *all* files
affect platform:android except those which have AFFECTS_ONLY for some
other platform. So `stylo/foo` would have AFFECTS set to
['platform:stylo'], but `build/foo` would have ['platform:android',
'platform:desktop', 'platform:stylo'].
The details of implementing this efficiently are just that -- details.
Note that this gets us the necessary condition that every file affects
*something*, avoiding the case where a push to a particular file might
trigger no tasks.
So let's bikeshed on this - both the try division and the "affects"
stuff - a little, and then we can re-evaluate the level of agreement
and see if we should sit down in SFO.
Dustin
[*] This is the 72nd meaning of the term "platform" - I'm open to better ideas
_______________________________________________
dev-builds mailing list
dev-builds@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-builds
