I'll only add that moving this to a separate file set would have the benefit of significantly simplifying the moz.build reading code, which is quite complex and somewhat slow. I don't have a profile offhand that points to Files() as a bottleneck, but it could become a concern if we were to continue extending it to be everything we want it to be.
Moving generally to moz.build files containing almost only things needed to execute the build would be a welcome trend as far as I'm concerned. Chris On Wed, Mar 21, 2018 at 12:33 AM, Dustin Mitchell <dus...@mozilla.com> wrote: > I agree that the fit with moz.build files is awkward. Other issues > (besides those tom raised) include: > - two different methods of reading files can give different results > - the method of combining different `with Files` clauses, even in > different files, is not clear > - *very* deep Python magic used in the implementation makes it > difficult to understand or modify (and I'm no Python n00b) > > We haven't gotten very far with SCHEDULES partly because it's not > clear (to me or anyone) how to express more fine-grained components in > moz.build files. > > I'd be happy with another solution that provides better expressiveness > and clearer semantics, whatever form that takes. > > Dustin > > > On Tue, Mar 20, 2018 at 1:44 PM, Gregory Szorc <g...@mozilla.com> wrote: > > moz.build files have a mechanism for associating metadata with files > [1]. We > > use the "with Files()" primitive [2] for: > > > > * Associating files with Bugzilla components > > * Defining how files impact certain CI components > > > > In the future, I fully anticipate us wanting to have a primitive that > allows > > us to associate files with modules and/or reviewers. This would enable > tools > > to automatically pick reviewer(s) for files that changed in a commit, for > > example. > > > > Putting this files-based metadata in moz.build files seemed like the > correct > > decision initially. moz.build files existed and it seemed better to > reuse an > > existing solution in the same space than to invent a new wheel. > > > > However, some deep cracks have formed that are causing me to question > > whether continuing to shoehorn this metadata into moz.build files is the > > right way to go. > > > > "If you build it they will come." The mapping of files to bug components > is > > valuable and people wanted an easy way to query it. So, there's an > endpoint > > on hg.mozilla.org that allows callers to specify a set of filenames and > > resolve the moz.build Files() metadata [3]. Because moz.build files are > > Python files, this involves running a heavily sandboxed process on the > > server with a snapshot of the Python code for evaluating moz.build > files. A > > big problem with this is that this Python code can only evaluate > moz.build > > files that were valid at the time the code snapshot was made. So when we > > upgrade the Python code, we may be unable to read moz.build files on > older > > revisions. And when we add new primitives to moz.build files, the old > Python > > code may choke reading moz.build files from newer revisions. It's a giant > > mess and is causing problems with automated tooling that needs moz.build > > Files metadata lookups to "just work." > > > > In bug 1402010, I wrote some PoC patches that change moz.build > evaluation of > > Files metadata to strip the Python AST of all nodes that aren't relevant > to > > "with Files()" blocks. This drastically reduces the surface area of > things > > that can cause moz.build evaluation to fail when using a snapshot of the > > Python code for evaluating moz.build files. While that solution does > work, I > > have my doubts that it is the right step forward. For one, you lose some > of > > the Python niceties, such as the ability to reference a variable or > > higher-level primitive when using "with Files()." For example, you may > want > > to iterate over a list of patterns to define common metadata. You could > no > > longer do that if the AST is rewritten in Files evaluation mode. That > > action-at-a-distance would be confusing to people editing moz.build > files. > > > > Anyway, I'm really tempted to declare the shoehorning of files-based > > metadata that isn't essential to the build system into moz.build files > to be > > a failed experiment. I'm tempted to say we should extract this static > data > > that is non-essential to builds into standalone data files (e.g. YAML > > files). Essentially, we would invent a new data format for representing > > files-based metadata and then port moz.build Files() blocks - namely > those > > declaring BUG_COMPONENT - to the new format. We would update tooling that > > evaluates files-based metadata to consume the new files. The new format > > would strive to be backwards and forwards compatible so that anyone with > a > > snapshot of the code for evaluating the files would be able to evaluate > very > > old and newer files to some degree. We may or may not keep Files() in > > moz.build files. The CI scheduling primitives in there are arguably the > > domain of the build system and should continue to live in moz.build > files. > > But we could have that discussion. > > > > This post is basically me saying "I'm not sure what we should do." I > still > > like the idea of shoehorning things into moz.build files because it is > > convenient and doesn't fragment where such knowledge is defined. But at > the > > same time, that is technically difficult to support and that difficulty > is > > actively causing pain due to existing solutions being brittle. > > > > I'm curious how others feel about this issue. And if we wanted to talk > about > > what a standalone file would look like, be named, etc, I think that > would be > > a constructive exercise. > > > > [1] > > https://firefox-source-docs.mozilla.org/build/buildsystem/ > files-metadata.html > > [2] > > https://firefox-source-docs.mozilla.org/build/buildsystem/ > mozbuild-symbols.html#sub-context-files > > [3] > > https://mozilla-version-control-tools.readthedocs.io/ > en/latest/hgmo/mozbuildinfo.html > _______________________________________________ > dev-builds mailing list > dev-builds@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-builds >
_______________________________________________ dev-builds mailing list dev-builds@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-builds
