Mike, et al,
I wanted to let you know that I have rebased my build-splendid branch on
top of current mozilla-central and have refactored it to work with mach.
The branch is still at [1].
All the code now lives under build/pylib/mozbuild. What's left of the
old code lives in build/pylib/mozbuild/mozbuild/buildconfig. That
directory also holds most new functionality not being tracked in bug
751795 because I'm trying to keep the patches separate so it is easier
to uplift them when the time comes. I've also checked in a custom
version of pymake to build/pylib/pymake which contains necessary patches
that haven't made it into upstream yet.
I've done a lot of refactoring as part of the rebase. frontend.py [2]
contains the bits that assemble the set of input/frontend files for the
build system. mozillamakefile.py [3] contains code for converting
Makefile.in's into generic data structures representing metadata from
them (e.g. the set of IDL files, files to export, etc).
The most important change is the introduction of a new concept:
backends. A backend is effectively an interface that takes a frontend
instance and can generate() backend files/state which can be used to
build() the tree. The LegacyBackend [4] implements Mozilla's existing
build backend: it converts Makefile.in's to Makefile's and runs make.
The HybridMakeBackend [5] is much more exciting. It takes the extracted
data from Makefile.in's (using MozillaMakefile.get_data_objects), writes
out a non-recursive .mk file corresponding to each Makefile.in, and
strips variables from the Makefile related to actions now performed by
the non-recursive make files. During building, it executes both the old
(slow) non-recursive make system and the new (insanely fast) recursive
make system. Let me say that again: *I have non-recursive make working
in mozilla-central in harmony with the existing recursive make backend*
(for some functionality, anyway). The export tier works very well and
already shaves a few minutes off of build times. It also has true no-op
builds that take less than 3s! I'm working on C++ compilation (current
implementation is very hacky, so please don't look at it).
Anyway, you could use HybridMakeBackend as a starting point for
implementing a TupBackend. If you want to do that, you'll need some
plumbing in the CLI [6].
Currently, the workflow is suboptimal:
$ ./mach configure
$ ./mach buildbuild hybridmake
(or |./mach buildconfig hybridmake| to just do generation without building).
Mike Hommey has plans to make configure support customizing the build
backend. For now, we just have configure do it's default "output
Makefile's" behavior then patch over it inside the generate() of the
build splendid backend.
There are many parts still very alpha and there is a lot of missing
functionality (like intelligently detecting when a new backend
generation is required). But, I think things are stable enough for you
to play around with. I'm pretty sure you have enough to create a working
Tup backend (for some functionality, anyway).
For people wanting to use this, I recommend not attempting to use it
just yet. It is still broken in a few major areas. I plan to expose a
standalone patch that one can just |hg qimport| or |git apply| once I'm
comfortable with people testing this as part of their day-to-day work.
And, yes, I'm attempting to upstream patches so this can eventually land
in m-c.
Gregory
[1] https://github.com/indygreg/mozilla-central/tree/build-splendid
[2]
https://github.com/indygreg/mozilla-central/blob/build-splendid/build/pylib/mozbuild/mozbuild/buildconfig/frontend.py
[3]
https://github.com/indygreg/mozilla-central/blob/build-splendid/build/pylib/mozbuild/mozbuild/buildconfig/mozillamakefile.py
[4]
https://github.com/indygreg/mozilla-central/blob/build-splendid/build/pylib/mozbuild/mozbuild/buildconfig/backend/legacy.py
[5]
https://github.com/indygreg/mozilla-central/blob/build-splendid/build/pylib/mozbuild/mozbuild/buildconfig/backend/hybridmake.py
[6]
https://github.com/indygreg/mozilla-central/blob/build-splendid/build/pylib/mozbuild/mozbuild/cli/buildconfig.py
On 7/16/12 8:56 PM, Mike Shal wrote:
Hi Gregory,
On Mon, Jul 16, 2012 at 11:42 PM, Gregory Szorc <g...@mozilla.com> wrote:
On 7/16/12 8:14 PM, Mike Shal wrote:
If not, we either need to change how things are defined in our
Makefile.in's
or we introduce an abstraction layer that generates Tupfiles. I was
operating on the assumption we'd have to go with the latter.
I don't think it makes sense to change Makefile.in's just to support
tup at this time (aside from those fixes you have already logged as
bugs), unless a small fix to a Makefile.in would save a lot of re-work
in a Tupfile or something.
We are changing our Makefile.in's to support N build systems, not just tup.
Righto :)
The general plan of attack is to provide some autoconf glue that allows
different "generators" to be used. By default, we'll go with the existing
straight conversion (Makefile.in -> Makefile). But, you could easily swap in
a Makefile.in -> Tupfile converter.
As I understand things (and Mike Hommey should confirm this), the set of
build configuration/input files will be saved to config.status along with
whatever build backend generator script was passed into configure. When
config.status runs, it will invoke the generator script, which will
write/update the build backend files appropriately.
We /think/ this is generic enough to work with any build backend. Is this
sufficient for tup? If not, what do we need to change?
That sounds reasonable to me - I don't think there would be a problem
with this for tup.
I also like the abstraction layer because that means we have a data
structure representing our build system definition. And, we can
theoretically convert that into Visual Studio, Xcode, etc files.
Do you consider static data-driven Makefile.in's an abstraction layer?
Or are you referring to something else?
I imagine that at some point we'll probably have code in the tree that
parses Makefile.in's (or whatever) into some monolithic data structure and
provides APIs for inspecting that. This will probably be along the lines of
extractor.py from my build-splendid branch. This solution will abstract away
the existence of Makefile.in's and allow build backends to consume Python
rather than having to worry about consuming the source. Think of it as an
API to the build configuration. All build backend generators would likely
use this API because it makes sense to do so (to avoid wheel reinvention).
Ahh, I see now. Thanks for the feedback!
-Mike
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
