moz.build files have a mechanism for associating metadata with files [1]. We use the "with Files()" primitive [2] for:
* Associating files with Bugzilla components * Defining how files impact certain CI components In the future, I fully anticipate us wanting to have a primitive that allows us to associate files with modules and/or reviewers. This would enable tools to automatically pick reviewer(s) for files that changed in a commit, for example. Putting this files-based metadata in moz.build files seemed like the correct decision initially. moz.build files existed and it seemed better to reuse an existing solution in the same space than to invent a new wheel. However, some deep cracks have formed that are causing me to question whether continuing to shoehorn this metadata into moz.build files is the right way to go. "If you build it they will come." The mapping of files to bug components is valuable and people wanted an easy way to query it. So, there's an endpoint on hg.mozilla.org that allows callers to specify a set of filenames and resolve the moz.build Files() metadata [3]. Because moz.build files are Python files, this involves running a heavily sandboxed process on the server with a snapshot of the Python code for evaluating moz.build files. A big problem with this is that this Python code can only evaluate moz.build files that were valid at the time the code snapshot was made. So when we upgrade the Python code, we may be unable to read moz.build files on older revisions. And when we add new primitives to moz.build files, the old Python code may choke reading moz.build files from newer revisions. It's a giant mess and is causing problems with automated tooling that needs moz.build Files metadata lookups to "just work." In bug 1402010, I wrote some PoC patches that change moz.build evaluation of Files metadata to strip the Python AST of all nodes that aren't relevant to "with Files()" blocks. This drastically reduces the surface area of things that can cause moz.build evaluation to fail when using a snapshot of the Python code for evaluating moz.build files. While that solution does work, I have my doubts that it is the right step forward. For one, you lose some of the Python niceties, such as the ability to reference a variable or higher-level primitive when using "with Files()." For example, you may want to iterate over a list of patterns to define common metadata. You could no longer do that if the AST is rewritten in Files evaluation mode. That action-at-a-distance would be confusing to people editing moz.build files. Anyway, I'm really tempted to declare the shoehorning of files-based metadata that isn't essential to the build system into moz.build files to be a failed experiment. I'm tempted to say we should extract this static data that is non-essential to builds into standalone data files (e.g. YAML files). Essentially, we would invent a new data format for representing files-based metadata and then port moz.build Files() blocks - namely those declaring BUG_COMPONENT - to the new format. We would update tooling that evaluates files-based metadata to consume the new files. The new format would strive to be backwards and forwards compatible so that anyone with a snapshot of the code for evaluating the files would be able to evaluate very old and newer files to some degree. We may or may not keep Files() in moz.build files. The CI scheduling primitives in there are arguably the domain of the build system and should continue to live in moz.build files. But we could have that discussion. This post is basically me saying "I'm not sure what we should do." I still like the idea of shoehorning things into moz.build files because it is convenient and doesn't fragment where such knowledge is defined. But at the same time, that is technically difficult to support and that difficulty is actively causing pain due to existing solutions being brittle. I'm curious how others feel about this issue. And if we wanted to talk about what a standalone file would look like, be named, etc, I think that would be a constructive exercise. [1] https://firefox-source-docs.mozilla.org/build/buildsystem/files-metadata.html [2] https://firefox-source-docs.mozilla.org/build/buildsystem/mozbuild-symbols.html#sub-context-files [3] https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmo/mozbuildinfo.html
_______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
