We added a capability to the Firefox build system that will ultimately
lead to speeding up the build and we need your help to realize its
potential.
Background
==========
The Firefox build executes as a pipeline of stages called tiers. The
tiers are currently export -> compile -> misc -> libs -> tools. The
build system iterates the tiers and builds directories in those tiers.
The export, compile, and misc tiers are mostly concurrent. If you
execute with |make -j32| or more, they should saturate your cores. If
you have access to a modern, multi-core machine, you realize the
benefits of that machine.
The libs and tools tier, by contrast, aren't concurrent. This is mostly
due to historical reasons. Back in the day, the traversal order of the
directories was hard-coded by the order directories were registered in
(DIRS += foo bar). If you had a cross-directory dependency, you
registered one directory before the other.
Furthermore, back when we only had export, libs, and tools tier, these
tiers were conceptually "pre-build," "build," and "post-build" tiers.
libs became a dumping ground for most build tasks. Fast forward a few
years and libs is still where most of the one-off build rules live. libs
still executes in the order directories are defined in.
Mission Misc
============
The libs tier slows down the build because it isn't executed
concurrently. By how much? As always, "it depends." But on my machine,
traversing the libs tier on a no-op build takes ~50% of the wall time of
the build (~22s/45s). The inefficiency of the libs tier is even more
pronounced during a full build (multi-core usage drops off a cliff).
*The libs tier is clownshoes.*
Our goal is to slowly kill the libs tier by moving things elsewhere,
where they can execute with much higher concurrency and where
cross-directory dependencies are properly captured. That will leave us
with a more efficient build system that executes faster.
This is where we need your help.
The tree has tons of build tasks that run during the libs tier. Many of
them have no dependencies and can be safely moved into the misc tier.
*We need your help writing patches to move things to the misc tier.*
How You Can Help
================
1) Search for build rules in your Makefile.in that execute in the libs
tier/target in your Makefile.in files
2) Examine the dependencies
3) If it doesn't depend on anything in the libs tier/target or doesn't
depend on things outside the current directory, it is probably safe to
move it to misc. (See also the warnings below.)
5) Try converting the rule to the misc tier. Add |HAS_MISC_RULE = True|
to the moz.build in that directory.
6) Test the build locally and push it to try
7) File a bug blocking bug 1099232. Initiate a code review and flag a
build peer (glandium, mshal, gps are preferred) for review.
For an example conversion, see
https://hg.mozilla.org/mozilla-central/rev/b1a9e41d3f4b
For a log showing what all gets built in libs, see
https://people.mozilla.org/~gszorc/libs.log (this is an OS X build). We
ideally want this log to be empty.
INSTALL_TARGETS and PP_TARGETS
------------------------------
The default rule/tier for INSTALL_TARGETS and PP_TARGETS is libs. Unless
you explicitly declare a *_TARGET variable, the rule will execute during
libs and will be slower.
*Merely grepping for "libs" is thus not sufficient for identifying rules
that execute during libs.*
Here is how you move an INSTALL_TARGETS rule from libs to misc.
Before:
foo_FILES := foo.js
foo_DEST := $(DIST)/bin
INSTALL_TARGETS += foo
After:
foo_FILES := foo.js
foo_DEST := $(DIST)/bin
foo_TARGET := misc
INSTALL_TARGETS += foo
Things to Watch Out For
-----------------------
When evaluating whether you can convert something from libs to misc:
* Stay away from things referencing $(XPI_NAME)
* Stay away from things related to JAR manifest parsing
* Stay away from things related to packaging
* Stay away from things related to l10n
There be dragons around all of these things. It is best to let a build
peer handle it.
If you aren't sure, don't hesitate to ask a build peer. We (the build
peers) want help tackling low-hanging fruit: don't waste your time on
something beyond your skill level: you likely won't derive anything but
pain. If it doesn't work right away, seriously consider moving on and
punting to someone with more domain knowledge.
Cross-Directory Dependencies
----------------------------
Did you find a rule that depends on another directory? For things in
moz.build, this should "just work." But for Makefile.in rules, you need
to explicitly declare that cross-directory dependency or the build is
susceptible to race conditions.
To declare a cross-directory dependency for the misc tier, you'll want
to add an entry at the bottom of config/recurse.mk.
For example, say you have a misc rule in "xpcom/system" that depends on
a misc rule in "xpcom/base." You would add the following to recurse.mk:
xpcom/system/misc: xpcom/base/misc
The build system integrates this into the traversal logic to ensure that
xpcom/base is traversed before xpcom/system is evaluated.
Conclusion
----------
If you have any questions, ask in #build.
I look forward to seeing many review requests in my inbox!
Gregory
_______________________________________________
dev-builds mailing list
dev-builds@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-builds
