Greg, thanks for starting this discussion. On Tue, Jan 23, 2018 at 4:34 PM, Gregory Szorc <g...@mozilla.com> wrote:
> Speaking as a maintainer of the Firefox build system, we try to be > conservative about the set of dependencies required to build Firefox from > source. We recognize that every required dependency is a potential source > of failure and complexity. Dependencies can create complications for > packagers. From a build system perspective, it's in our best interest to > minimize the dependencies required to build Firefox. > > Our desire to keep things simple can be at odds with the wishes of Firefox > developers who wish to leverage new and exciting tools and technologies. In > short, the policy of minimizing build dependencies can externalize costs > onto overall Firefox development by hindering people from using better > tools. This adversely affects the development velocity of Firefox and can > cause product quality to suffer. > > For a few years now, various pockets of Firefox development have wanted to > use Node.js as part of the development workflow. I wouldn't say they > directly want to use Node.js: they want to use the large ecosystem of tools > built around Node.js. But that's splitting hairs. Our build system policy > has been that leveraging Node.js and its ecosystem for supplemental > workflows is fine, but shipping a build dependency on Node.js is not. Many > Firefox developers are now using Node.js in their day-to-day workflow for > things like running ESLint. We're using Node.js in CI. But we're not > forcing people to have Node.js installed to build Firefox. > I got interested in "paving Node.js/NPM cowpaths" in existing projects that are in some way in mozilla-central a little while ago. I started interviewing people and teams in such projects, and have produced an "ecosystem matrix" of what such projects are doing at https://docs.google.com/document/d/1beGX-xA3qtZQme-JKa2gOgyXo-GPdEXbapeYO3dY858/ (That document is supposed to be open for comment to anyone with the link; if it's not, please let me know!) I hope this matrix can help inform this discussion by illuminating what's actually in use right now. That document also includes notes from my interviews (with the permission of the interview subjects). > Various groups have routed around the limitation that Node.js can't be > required to build Firefox. There are now Firefox features that do require > Node.js to build. However, the output from the Node.js tools is checked > into the Firefox source repository. So from the perspective of the Firefox > build system, Node.js doesn't exist and therefore isn't a build dependency. > In the dependency matrix above, the projects that are doing this are the projects using WebPack. > The status quo is not ideal. The more people I speak with, the more > apparent it is that our current policy of not allowing Node.js tooling in > the build system is causing more problems than it is preventing. Speaking > as the build system module owner and someone who cares about developer > workflows, tooling, and developer productivity, I don't think the current > policy is good for Mozilla. > > I'd like to start a discussion about requiring Node.js to build Firefox. > > What do I mean by "require Node.js?" Let's assume I mean having a usable > Node.js executable on the host system to be used during a Firefox build. > > What about npm or a package manager? I would strongly prefer to limit the > required dependency to Node.js itself. While the Firefox build system would > depend on 3rd party packages and tools (such as Babel), I'm pretty > insistent (as a build system maintainer) that these dependencies be > vendored into the Firefox source repository so as to not incur a run-time > dependency on a packaging service. I've seen the chaos that "left-pad" > caused. I don't fully trust the security model of JavaScript package > distribution. I don't think we can risk the ability to build Firefox or the > integrity of the Firefox product by the availability and integrity of a 3rd > party packaging service. That may sound like a harsh thing to say. But it's > the posture we've applied elsewhere (such as to Python packages and PyPI). > So, this means that all JavaScript executed by Node.js as part of the build > would either be provided by Node.js itself or the Firefox source > repository. If we needed to use a package manager as part of the build, > that package manager could be vendored in the Firefox repository along with > other JavaScript libraries (not unlike how we currently vendor Python's pip > package manager). > I whole-heartedly agree that vendoring required code -- that is, things that must be NPM `dependencies` -- into mozilla-central is required for our release management and product integrity efforts. It's my belief that vendoring the NPM-based *tooling* -- that is, things that could be NPM `devDependencies` -- that Firefox developers are actively using right now would be impractical. Just the node_modules from the testing configuration of our ES lint plugin (eslint-plugin-{mozilla,spidermonkey}) is roughly 150 NPM packages totaling 5000 files. We might have, say, 500-1000 NPM packages to vendor into mozilla-central to support the existing toolchains of the 6 NPM-using projects in the tree. Some of those packages probably require multiple versions. Updating this vendored sprawl of code that *doesn't ship in Firefox directly* seems like a waste of time and effort. Yes, we could add tooling to make this less painful, but I don't think we should eat that cost; and any layer of tooling to handle this vendoring will make all subsequent efforts more complex, and lead us to not update the vendored code. That in turn will lead us to not "follow the ecosystem" and make it less likely that we'll use the best tool for the job at hand. If we really want to avoid a build time dependency on NPM, then I think we should think through a small set of build system primitives that we want to support and not really think about NPM-based tooling at all. That is, can we support Babel, WebPack, ES lint, and perhaps Flow, for arbitrary NPM packages checked into the tree? (And vendor those hard dependencies into the tree.) If we can do that, we might even be able to stretch to running those _limited_ dependencies in SpiderMonkey with the help of WebPack or browserify or some other Node.js-in-the-browser conversion, plus special sauce for SpiderMonkey and our build environment. But that leaves us not using the NPM unit testing ecosystem (Mocha, Chai, Sinon, Jasmine, Jest, etc), and not using the code coverage ecosystem, and not using the editor integration ecosystem (natively), etc, etc. On an entirely different front, Mozilla has some experience with requiring a package service at build time. Firefox for Android requires third-party Maven repositories (BinTray's jcenter and Google's Android repository) at build-time. We have a two-stage build process in automation that collects and caches dependencies, to isolate us from third-party outages. Obviously Firefox for Android is a small project compared to Gecko and Firefox for Desktop, but early experience has been very positive. And using Android Studio and the features of the Android-Gradle build plugin has been a phenomenal productivity win for the Firefox for Android teams; has enabled proper unit testing and static analysis of the Firefox for Android code base; and has allowed GeckoView to architect itself as a separable component of Firefox for Android, which is critical for the long-term success of GeckoView. A few people at Mozilla have poked at this problem already. We have a > general sense of where some pain points for us will be. We know that > getting modern versions of Node.js installed on various distributions > requires using 3rd party package repositories. We know that Windows support > could be painful. We know that installing common packages can result of > dozens if not hundreds of dependencies being added. We know this could lead > to us having to install thousands of files as part of the Firefox build - > an overhead I'm not keen on seeing. We know all of this can add up to a > significant amount of overhead to support. (Yet it still feels like a > lesser problem than having people work around not being able to use Node.js > directly.) > > What we don't generally know is the impact requiring Node.js would have on > downstream packagers. Our adoption of Rust last year was a long and > sometimes painful process. I have a feeling that requiring Node.js would be > a similar experience. But like Rust, I feel that adopting Node.js is in the > best long-term interest for Firefox development velocity and product > quality. I'm reluctant to cause more hardship by introducing a new build > dependency. But it's very difficult to keep saying we can't use Node.js in > the Firefox build system. I wish I could say "we'll build SpiderMonkey and > use that instead." Unfortunately, many Node.js tools don't work with > SpiderMonkey, so that's not an option. Plus there are difficulties with > cross-compilation. As sad as it makes me to say it, SpiderMonkey is not an > option: Node.js is the only viable option. > Even though I floated an alternative above (browserify and labour), I concur that Node.js is the only viable option. > If we require Node.js to build Firefox, what are the requirements, > desires, and hardships of downstream packagers and consumers of the Firefox > build system? Keep in mind that mozilla-central right now is Firefox 60. > That will become ESR 60 in May. > I wish I could say more about downstream packagers but I have nothing to add here. Thanks again! Long live the glorious web development ecosystem! Nick
_______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
