Hi, Following is a proposal for short term wins for Firefox front end developers.
The starting point is that there are too many things to change in the recursive make build system to allow for something like this to work properly for many different use cases. This also starts from the fact that, well, we've kept the recursive make build system use the same model it's been using for, well, close to 20 years. That model is flawed, and I thought it was about time to do something about it, while addressing real needs at the same time. My proposal is to start a new build backend from scratch, with the aim to grow it into a complete solution. Build backend means I'm not talking about replacing moz.build files with something new. I'm talking about the build system they're munged into. It would initially piggy back on the existing build system, because there are things that aren't exposed in moz.build yet that are necessary. It would also, initially, not care about the entire build, and even less about things that are outside of Firefox (tests, addons, etc.) or compilation. My initial work is this patch: https://pastebin.mozilla.org/8846539 (Note: it requires the patches from bug 1204712, bug 1204719 and bug 1204715, all landed on mozilla-inbound, as of writing) What it allows is to: - Run `mach configure` with a mozconfig with ac_add_options --disable-compile-environment - Download and unpack a nightly - Use toolkit/mozapps/installer/unpack.py to explode its omni.ja files - Move that unpacked nightly in $objdir/dist/bin (for mac, that involves more fiddling, because dist/bin is a somewhat flattened version of the .app directory) - Ensure the files in $objdir/dist/bin are older than the source files. - Run `mach build-backend -b FasterMake` - Run `mach build faster` After the last command, $objdir/dist/bin should contain a bastardized Firefox, with xul, js, etc. coming from the source tree, and the remainder still being there from the original nightly. `mach run` should work with that. Note: removing files is not supported in this initial implementation, so removing files will still require manual intervention. Editing files #include'd from others should work properly. >From there, changes to js, xul, images, etc. only require running `mach build faster`, and on my machine, that takes something between 3 and 4 seconds. Future iterations should allow subsecond updates. Other changes would require a full `mach build`, without --disable-compile-environment. Anyways, as you can see with the list of steps above, this requires a lot of manual work at the moment, so my proposal extends to streamlining this process. - Extend mach artifact to handle any type of Firefox build (not only Firefox for Android), and to unpack builds completely (not just .so files), and doubly (by which I mean, unpacking omni.ja as well, with the code used by toolkit/mozapps/installer/unpack.py) - Extend the build system to allow to apply multiple backends at the same time. `mach build-backend -b FasterMake` only spends 0.05s on my machine generating its build system, but still spends 4s that are already spent during configure to read and analyze moz.build files. That's also true of other backends, and it would be better to allow people to add their prefered extra backends to their mozconfig. (and they all rely on the recursive make backend, so that one can be kept always on for now). It would also help make some of those extra backends enabled by default depending on what's built. - Ensure --disable-compile-environment works with all the above, but I don't think there is actually anything to do for this. As a matter of fact, I've tested it, and it seems to work well enough (and that lead me to recently fix bug 1203851) - Make mac builds fill dist/Nightly.app directly instead of filling dist/bin (or fill dist/bin as if it were dist/Nightly.app), because having to run all of https://dxr.mozilla.org/mozilla-central/source/browser/app/Makefile.in?offset=0#95-109 at the end of every build is a pain point. This would have benefits for other kinds of builds, too. With those four above adressed, we can have a better experience for Firefox frontend developers in a very short time. The last point is likely to take some time, so it might make sense to replace it with an extension to the first point, where mach artifact would flatten the build to make it look like dist/bin, and the FasterMake backend would invoke the right rules from browser/app/Makefile.in. The only downside I can see to this plan is that since the new backend would not be exercized on automation, it would have chances of breaking. Fair enough, but the same can be said of all the other backends. On the other hand, I can see many benefits: - The new backend would not be exercized on automation. While it can be seen as a downside, it is also a *huge* benefit, because it breaks all the limits that come from ensuring all of automation works. Not having to care about l10n is what makes the PoC work so well. - As mentioned above, short term win for frontend developers - Other use cases can plug on top of it. We could quickly add tests, for example, and this is where the piggy-backing happens: we can just invoke small parts of the current build system when it makes sense. - It sets things in motion so that a future with non-make based build backends can actually happen (like, replacing make with ninja or tup in this new backend would be fairly easy). - It allowed me to spot some of the shortcomings of the current moz.build model with the emitter and helped get a clearer vision of where we should be going on the long term (we can discuss this later). See for instance the horror show in FasterMakeBackend.consume_object. 6 slithly different ways to be notified of files to install or preprocess, and that's only for things relevant to Firefox. The almost 1:1 mapping between moz.build variables and emitted objects is not helpful. I'm almost ready to send that patch to review. At this point, I just want to add a few comments to fastermake.py before doing so. Thoughts on the overall proposal? Mike _______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
