I talked to gps today and told him I would let him know my numbers on my machine. I will share them with everybody:
My Win32 debug clobber build (mach build after mach clobber) was 39:54.84 today, up from ~33:00 a few months ago. Not sure if it is my system. Immediate rebuild (no-op) was 10:13.05. A second no-op rebuild was 10:32.36. It looks like every shared library and every executable got relinked. This is on Windows7 64-bit, with 32GB of memory, an Intel 520 SSD, and an Intel i7-3920XM @ 2.90Ghz/3.10Ghs. (Lenovo W530, plugged in, on "Maximum Performance" power setting). I was doing other things (browsing, coding) in the foreground while these builds ran in the background. Cheers, Brian On Fri, Aug 2, 2013 at 2:13 PM, Gregory Szorc <g...@mozilla.com> wrote: > (Cross posting. Please reply to dev.builds.) > > I've noticed an increase in the number of complaints about the build > system recently. I'm not surprised. Building mozilla-central has gotten > noticeably slower. More on that below. But first, a request. > > Many of the complaints I've heard have been from overhearing hallway > conversations, noticing non-directed complaints on IRC, having 3rd parties > report anecdotes, etc. *Please, please, please voice your complaints > directly at me and the build peers.* Indirectly complaining isn't a very > effective way to get attention or to spur action. I recommend posting to > dev.builds so complaints and responses are public and easily archived. If > you want a more personal conversation, just get in contact with me and I'll > be happy to explain things. > > Anyway, on to the concerns. > > Builds are getting slower. > http://brasstacks.mozilla.com/**gofaster/#/<http://brasstacks.mozilla.com/gofaster/#/>has > high-level trends for our automation infrastructure. I've also noticed > my personal machines taking ~2x longer than they did 2 years ago. > Unfortunately, I can't give you a precise breakdown over where the > increases have been because we don't do a very good job of recording these > things. This is one reason why we have better monitoring on our Q3 goals > list. > > Now, on to the reasons why builds are getting slower. > > # We're adding new code at a significant rate. > > Here is a breakdown of source file types in the tree by Gecko version. > These are file types that are directly compiled or go through code > generation to create a compiled file. > > Gecko 7: 3359 C++, 1952 C, 544 CC, 1258 XPIDL, 110 MM, 195 IPDL > Gecko 14: 3980 C++, 2345 C, 575 CC, 1268 XPIDL, 272 MM, 197 IPDL, 30 WebIDL > Gecko 21: 4606 C++, 2831 C, 1392 CC, 1295 XPIDL, 292 MM, 228 IPDL, 231 > WebIDL > Gecko 25: 5211 C++, 3029 C, 1427 CC, 1268 XPIDL, 262 MM, 234 IPDL, 441 > WebIDL > > That nets totals of: > > 7: 7418 > 14: 8667 > 21: 10875 > 25: 11872 > > As you can see, we're steadily adding new source code files to the tree. > mozilla-central today has 60% more source files than Gecko 7! If you assume > number of source files is a rough approximation for compile time, it's > obvious why builds are getting slower: we're building more. > > As large new browser features like WebRTC and the ECMAScript > Internationalization API continue to dump hundreds of new source files in > the tree, build times will increase. There's nothing we can do about this > short of freezing browser features. That's not going to happen. > > # Header dependency hell > > We have hundreds of header files that are included in hundreds or even > thousands of other C++ files. Any time one of these widely-used headers > changes, the object files get invalidated by the build system dependencies > and we have to re-invoke the compiler. This also likely invalidates ccache, > so it's just like a clobber build. > > No matter what we do to the build backend to make clobber builds faster, > header dependency hell will continue to undermine this progress for > dependency builds. > > I don't believe the build config group is in a position to tackle header > dependency hell at this time. We are receptive to good ideas and will work > with people to land patches. Perhaps an ad-hoc group of Platform developers > can band together to address this? > > # Increased reliance on C++ language features > > I *suspect* that our increased reliance on C++ language features such as > templates and new C++11 features is contributing to slower build times. > It's been long known that templates and other advanced language features > can "blow up" the compiler if used in certain ways. I also suspect that > modern C++11 features haven't been optimized to the extent years-old C++ > features have been. Combine this with the fact compilers are working harder > than ever to optimize code and it wouldn't surprise me if a CPU cycle > invested in the compiler isn't giving the returns it used to. > > I would absolutely love for a compiler wizard to sit down and "profile" > Gecko C++ in Clang, GCC, and MSVC. If there are things we can do to our > source or to the compilers themselves to make things faster, that could be > a huge win. > > Like dependency hell, I don't believe the build config group will tackle > this any time soon. > > # Clobbers are more frequent and more annoying > > Clobbers are annoying. It annoys me every time I see the CLOBBER file has > been updated. I won't make excuses for open bugs on known required-clobber > issues: we should fix them all. > > I suspect clobbers have become more annoying in recent months because > overall build times have increased. If builds only took 5 minutes, I'm not > sure the cries would be as loud. That's no excuse for not fixing it, > however. Please continue to loudly complain every time there is a clobber. > > # Slowness Summary > > There are many factors contributing to making the build system slower. I > would argue that the primary contributors are not within the control of the > build config group. Instead, the fault lives with all the compiled code > (mainly C++) in the tree. I'm not trying to point fingers or deflect blame. > But if I had to give one answer for "why have the builds gotten slower," > that answer is "we're compiling a lot more code." > Since we can't change that, the next question becomes "what are we doing > to make building faster." The answer: lots of things. > > # Building faster > > One of our Q3 goals is to replace the "export tier" with something more > efficient. More on tiers at [1]. This should make builds faster, especially > on pymake. Just earlier this week we made WebIDL and XPIDL code generation > concurrent. Before, they executed serially, failing to utilize multiple CPU > cores. Next steps are XPIDL code gen, installing headers, and > preprocessing. This is all tracked in bug 892644. > > We're also slowly working towards moving all the data that describes how > to invoke the compiler into moz.build files. Once that data is there, we > will be able to experiment with precompiled headers and other compiler > techniques to make compilation faster. This will also enable us to put all > the compiler rules in a single make file. This will enable scaling out to > as many cores as you have available. I'm optimistic we'll have this by the > end of 2013. > > I concede that the rate of progress is slower than I would like. I would > halve the build times for everyone tomorrow if I could! Unfortunately, this > isn't an overnight project. It will take time. We're starting to turn the > corner and realize benefits of moving data to moz.build files. Real gains > should be seen over the next few weeks and months. Hopefully the gains will > be able to outpace the setbacks from all the new code landing in the tree. > Please stick with us. Until then, please continue to complain about the > build system. If you have time, please stop in #build and ask how you can > help. You might be surprised what you could do! > > [1] https://developer.mozilla.org/**en-US/docs/How_Mozilla%27s_** > build_system_works#Build_Tiers<https://developer.mozilla.org/en-US/docs/How_Mozilla%27s_build_system_works#Build_Tiers> > > (Please reply to dev.builds.) > ______________________________**_________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/**listinfo/dev-platform<https://lists.mozilla.org/listinfo/dev-platform> > -- Mozilla Networking/Crypto/Security (Necko/NSS/PSM) _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
