On 8/2/13 3:38 PM, Ehsan Akhgari wrote:
Hmm. I'm not sure if the number of source files is directly correlated
to build times, but yeah there's clearly a trend here!
I concede a lines of code count would be a better indicator. I'm lazy.
# Header dependency hell
I have been playing with an idea in my head about this. What if we had
a list of the most popular headers in our tree, and we looked through
them and tried to cut down the number of #includes in the headers? That
should help create more isolated sub-graphs and hopefully help with
breaking the most severe dependency chains.
Writing a tool to spit out this information should be fairly easy.
I'll try to get a tool in the tree for people to run.
https://bugzilla.mozilla.org/show_bug.cgi?id=901132
# Increased reliance on C++ language features
But I'm not convinced at all about the C++11 features contributing to
this. I cannot think of any reason at all why that should be the case
for the things that we've started to use. Do you have any evidence to
implicate some of those features?
No. Just my general distrust of new/young vs mature software.
# Clobbers are more frequent and more annoying
This should be relatively "easy" to address (compared to the other
things that we can do, of course). I assert that every time we touch
the CLOBBER file, it's because the build system could not figure out the
dependencies properly. Fortunately we can easily log the CLOBBER file
and go back in time and find all of the patches that included CLOBBER
modifications and debug the build dependency issues. Has there been any
effort to address these issues by looking at the testcases that we have
in form of patches?
To some degree, yes. https://bugzilla.mozilla.org/show_bug.cgi?id=890744
is a good example. Vacation schedules didn't align for quick action.
There may also be a pymake bug or two involved.
Also, you could say people have been touching CLOBBER prematurely. I
know there are a few cases where CLOBBER was touched in hopes it fixed a
problem, didn't, and the commit history was left with a changeset that
changed CLOBBER.
# Slowness Summary
Every time that we don't utilize 100% of our cores during the build
process, that's an unnecessary slowdown. That consistently wastes a lot
of time during every build, and it also means that we can't address this
by getting more powerful machines. :(
Right. If you plot CPU usage vs time, we can make the build faster by
"filling out the box" and using 100% of all cores or by decreasing the
total number of required CPU cycles to build. We have chosen to focus
mostly on the former because optimizing build actions can be a lot of
work. We've got lucky in some cases (e.g. WebIDLs in bug 861587). I fear
compiling C++ will be much harder. I'm hoping PCH and fixing dependency
hell are "medium-hanging" fruits.
I also have measurements that show we peak out at certain concurrency
levels. The trend in CPUs is towards more cores, not higher clock speed.
So focusing on effective core usage will continue to be important.
Derecursifying the build will allow us to use more cores because make
won't be starved during directory traversal. Remember, concurrent make
only works within the same directory or for directories under
PARALLEL_DIRS. Different top-level directories during tier traversal
(e.g. dom and xpcom) are executed sequentially.
# Building faster
One of our Q3 goals is to replace the "export tier" with something more
efficient. More on tiers at [1]. This should make builds faster,
especially on pymake. Just earlier this week we made WebIDL and XPIDL
code generation concurrent. Before, they executed serially, failing to
utilize multiple CPU cores. Next steps are XPIDL code gen, installing
headers, and preprocessing. This is all tracked in bug 892644.
Out of curiosity, why was the export tier the fist target for this? I
may lack context here, but the slowest tier that we have is the platform
libs tier. Wouldn't focusing on that have given us the biggest possible
bang for the buck?
Making platform libs faster will without a doubt have the biggest
impact. We chose to start with export first for a few reasons.
First, it's simple. We had to start somewhere. platform/libs is a
magnitude more complex. We are making major refactorings in export
already and we felt it best to prove out concepts with export rather
that going for the hardest problem first.
Second, export is mostly standalone targets. We would like to "port" the
build backend bottom up instead of top down so we can make the
dependencies right from the beginning. If we started with platform/lib,
we'd have to hack something together now and revamp it with proper
dependencies later.
Third, export is horribly inefficient. pymake spends an absurd amount of
time traversing directories, parsing make files and doing very little
for each directory in the export tier. Platform, by contrast, tends to
have longer-running jobs so cores aren't starved as often. When you look
at a graph of resource usage for the different tiers, this is obvious.
By removing export, we'll be removing a traversal from the tiers. This
could decrease no-op build times by as much as 30%!
Fourth, removing export actually makes libs more efficient!
Specifically, moving XPIDL code gen out of export and libs and into the
"precompile" phase/tier makes libs faster! I was surprised when I first
saw this. Essentially, the XPIDL codegen tasks during libs block their
dependent C++ compile targets from running. So by moving XPIDL out of
libs, C++ compilation starts immediately without having to wait on XPIDL.
Fifth, it allows simultaneous development on the build system. For
example, I'm currently spending a lot of time refactoring export while
joey and mshal are moving pieces related to libs to moz.build, paving
the road for work there. We largely don't get in each others' way.
We're also slowly working towards moving all the data that describes how
to invoke the compiler into moz.build files. Once that data is there, we
will be able to experiment with precompiled headers and other compiler
techniques to make compilation faster. This will also enable us to put
all the compiler rules in a single make file. This will enable scaling
out to as many cores as you have available. I'm optimistic we'll have
this by the end of 2013.
In your recent dev-builds post
<https://groups.google.com/forum/#!topic/mozilla.dev.builds/QkzCD8h9OjM>
you estimated that this conversion is going to take about another year.
Have your estimates changed since then?
I'm optimistic we can get better C++ compilation rules in place for
*some* part of the tree by EOY. Converting all the C++ will likely take
longer.
For those of us not following the Build Config component, can you please
list the steps that you're planning to go through in order to achieve
these gains? The things that I've heard people talk about is
derecursifying the export tier and the moz.build conversion, but it's
not clear to me what the prioritization looks like, and what do we
expect to achieve with each piece of work...
It's hard to attach numbers to expectations because we really won't know
what we're looking at until we have it.
The steps are effectively to derecursify bits from the bottom up in
terms of dependencies. i.e. build actions without dependencies get
converted first. Export tier is mostly simple file copying,
preprocessing, codegen, etc, so we started there. Doing it this way will
facilitate supporting Tup sooner since Tup will want to know about all
dependencies. We may cut some corners to optimize C++ compilation and
linking sooner because those have the biggest potential for wall time
wins, which is what we're ultimately after.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
