arames created this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.
arames edited the summary of this revision.
arames added reviewers: bruno, rsmith.
Herald added a subscriber: dexonsmith.
The behavior is controlled by the `-fprebuilt-implicit-modules` option, and
allows searching for implicit modules in the prebuilt module cache paths.
The current command-line options for prebuilt modules do not allow to easily
maintain and use multiple versions of modules. Both the producer and users of
prebuilt modules are required to know the relationships between compilation
options and module file paths. Using a particular version of a prebuilt module
requires passing a particular option on the command line (e.g.
`-fmodule-file=[<name>=]<file>` or `-fprebuilt-module-path=<directory>`).
However the compiler already knows how to distinguish and automatically locate
implicit modules. Hence this proposal to introduce the
`-fprebuilt-implicit-modules` option. When set, it enables searching for
implicit modules in the prebuilt module paths (specified via
`-fprebuilt-module-path`). To not modify existing behavior, this search takes
place after the standard search for prebuilt modules. If not
Here is a workflow illustrating how both the producer and consumer of prebuilt
modules would need to know what versions of prebuilt modules are available and
where they are located.
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo
-fmodules-cache-path=prebuilt_modules_v1 <config 1 options>
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo
-fmodules-cache-path=prebuilt_modules_v2 <config 2 options>
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo
-fmodules-cache-path=prebuilt_modules_v3 <config 3 options>
clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap
-fprebuilt-module-path=prebuilt_modules_v1 <config 1 options>
clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap <non-prebuilt
config options>
With prebuilt implicit modules, the producer can generate prebuilt modules as
usual, all in the same output directory. The same mechanisms as for implicit
modules take care of incorporating hashes in the path to distinguish between
module versions.
Note that we do not specify the output module filename, so `-o` implicit
modules are generated in the cache path `prebuilt_modules`.
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo
-fmodules-cache-path=prebuilt_modules <config 1 options>
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo
-fmodules-cache-path=prebuilt_modules <config 2 options>
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo
-fmodules-cache-path=prebuilt_modules <config 3 options>
The user can now simply enable prebuilt implicit modules and point to the
prebuilt modules cache. No need to "parse" command-line options to decide
what prebuilt modules (paths) to use.
clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap
-fprebuilt-module-path=prebuilt_modules -fprebuilt-implicit-modules <config 1
options>
clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap
-fprebuilt-module-path=prebuilt_modules -fprebuilt-implicit-modules
<non-prebuilt config options>
This is for example particularly useful in a use-case where compilation is
expensive, and the configurations expected to be used are predictable, but not
controlled by the producer of prebuilt modules. Modules for the set of
predictable configurations can be prebuilt, and using them does not require
"parsing" the configuration (command-line options).
Repository:
rG LLVM Github Monorepo
https://reviews.llvm.org/D68997
Files:
clang/docs/Modules.rst
clang/include/clang/Driver/Options.td
clang/include/clang/Lex/HeaderSearch.h
clang/include/clang/Lex/HeaderSearchOptions.h
clang/lib/Driver/ToolChains/Clang.cpp
clang/lib/Frontend/CompilerInstance.cpp
clang/lib/Frontend/CompilerInvocation.cpp
clang/lib/Lex/HeaderSearch.cpp
clang/lib/Serialization/ASTReader.cpp
clang/lib/Serialization/ASTWriter.cpp
clang/test/Modules/Inputs/prebuilt-implicit-module/a.h
clang/test/Modules/Inputs/prebuilt-implicit-module/module.modulemap
clang/test/Modules/prebuilt-implicit-modules.m
Index: clang/test/Modules/prebuilt-implicit-modules.m
===================================================================
--- /dev/null
+++ clang/test/Modules/prebuilt-implicit-modules.m
@@ -0,0 +1,27 @@
+// RUN: rm -rf %t
+// RUN: %clang_cc1 -x objective-c -fmodules %S/Inputs/prebuilt-implicit-module/module.modulemap -emit-module -fmodule-name=module_a -fmodules-cache-path=%t
+// RUN: find %t -name "module_a*.pcm" | grep module_a
+//
+// Check we use a prebuilt module when available, and do not build an implicit module.
+// RUN: rm -rf %t1
+// RUN: mkdir -p %t1
+// RUN: %clang_cc1 -x objective-c %s -I%S/Inputs/prebuilt-implicit-module -fmodules -fmodule-map-file=%S/Inputs/prebuilt-implicit-module/module.modulemap -fprebuilt-implicit-modules -fprebuilt-module-path=%t -fmodules-cache-path=%t1
+// RUN: find %t1 -name "module_a*.pcm" | not grep module_e
+//
+// Check that we correctly fall back to implicit modules if the prebuilt implicit module is not found.
+// RUN: %clang_cc1 -x objective-c %s -I%S/Inputs/prebuilt-implicit-module -fmodules -fmodule-map-file=%S/Inputs/prebuilt-implicit-module/module.modulemap -fprebuilt-implicit-modules -fprebuilt-module-path=%t -fmodules-cache-path=%t1 -fno-signed-char
+// RUN: find %t1 -name "module_a*.pcm" | grep module_a
+
+// Check that non-implicit prebuilt modules are always preferred to prebuilt implicit modules.
+// RUN: rm -rf %t2
+// RUN: mkdir -p %t2
+// RUN: %clang_cc1 -x objective-c -fmodules %S/Inputs/prebuilt-implicit-module/module.modulemap -emit-module -fmodule-name=module_a -fmodules-cache-path=%t
+// RUN: %clang_cc1 -x objective-c -fmodules %S/Inputs/prebuilt-implicit-module/module.modulemap -emit-module -fmodule-name=module_a -o %t/module_a.pcm -fno-signed-char
+// RUN: not %clang_cc1 -x objective-c %s -I%S/Inputs/prebuilt-implicit-module -fmodules -fmodule-map-file=%S/Inputs/prebuilt-implicit-module/module.modulemap -fprebuilt-implicit-modules -fprebuilt-module-path=%t -fmodules-cache-path=%t2
+// RUN: find %t2 -name "module_a*.pcm" | not grep module_a
+
+// expected-no-diagnostics
+@import module_a;
+int test() {
+ return a;
+}
Index: clang/test/Modules/Inputs/prebuilt-implicit-module/module.modulemap
===================================================================
--- /dev/null
+++ clang/test/Modules/Inputs/prebuilt-implicit-module/module.modulemap
@@ -0,0 +1 @@
+module module_a { header "a.h" }
Index: clang/test/Modules/Inputs/prebuilt-implicit-module/a.h
===================================================================
--- /dev/null
+++ clang/test/Modules/Inputs/prebuilt-implicit-module/a.h
@@ -0,0 +1 @@
+const int a = 1;
Index: clang/lib/Serialization/ASTWriter.cpp
===================================================================
--- clang/lib/Serialization/ASTWriter.cpp
+++ clang/lib/Serialization/ASTWriter.cpp
@@ -1675,6 +1675,7 @@
Record.push_back(HSOpts.DisableModuleHash);
Record.push_back(HSOpts.ImplicitModuleMaps);
Record.push_back(HSOpts.ModuleMapFileHomeIsCwd);
+ Record.push_back(HSOpts.EnablePrebuiltImplicitModules);
Record.push_back(HSOpts.UseBuiltinIncludes);
Record.push_back(HSOpts.UseStandardSystemIncludes);
Record.push_back(HSOpts.UseStandardCXXIncludes);
Index: clang/lib/Serialization/ASTReader.cpp
===================================================================
--- clang/lib/Serialization/ASTReader.cpp
+++ clang/lib/Serialization/ASTReader.cpp
@@ -5726,6 +5726,7 @@
HSOpts.DisableModuleHash = Record[Idx++];
HSOpts.ImplicitModuleMaps = Record[Idx++];
HSOpts.ModuleMapFileHomeIsCwd = Record[Idx++];
+ HSOpts.EnablePrebuiltImplicitModules = Record[Idx++];
HSOpts.UseBuiltinIncludes = Record[Idx++];
HSOpts.UseStandardSystemIncludes = Record[Idx++];
HSOpts.UseStandardCXXIncludes = Record[Idx++];
Index: clang/lib/Lex/HeaderSearch.cpp
===================================================================
--- clang/lib/Lex/HeaderSearch.cpp
+++ clang/lib/Lex/HeaderSearch.cpp
@@ -164,14 +164,41 @@
return {};
}
+std::string HeaderSearch::getPrebuiltImplicitModuleFileName(Module *Module) {
+ const FileEntry *ModuleMap =
+ getModuleMap().getModuleMapFileForUniquing(Module);
+ StringRef ModuleName = Module->Name;
+ StringRef ModuleMapPath = ModuleMap->getName();
+ StringRef ModuleCacheHash = HSOpts->DisableModuleHash
+ ? ""
+ : llvm::sys::path::stem(getModuleCachePath());
+ for (const std::string &Dir : HSOpts->PrebuiltModulePaths) {
+ SmallString<256> CachePath(Dir);
+ llvm::sys::fs::make_absolute(CachePath);
+ llvm::sys::path::append(CachePath, ModuleCacheHash);
+ std::string FileName =
+ getCachedModuleFileNameImpl(ModuleName, ModuleMapPath, CachePath);
+ if (!FileName.empty() && getFileMgr().getFile(FileName))
+ return FileName;
+ }
+ return {};
+}
+
std::string HeaderSearch::getCachedModuleFileName(StringRef ModuleName,
StringRef ModuleMapPath) {
+ return getCachedModuleFileNameImpl(ModuleName, ModuleMapPath,
+ getModuleCachePath());
+}
+
+std::string HeaderSearch::getCachedModuleFileNameImpl(StringRef ModuleName,
+ StringRef ModuleMapPath,
+ StringRef CachePath) {
// If we don't have a module cache path or aren't supposed to use one, we
// can't do anything.
- if (getModuleCachePath().empty())
+ if (CachePath.empty())
return {};
- SmallString<256> Result(getModuleCachePath());
+ SmallString<256> Result(CachePath);
llvm::sys::fs::make_absolute(Result);
if (HSOpts->DisableModuleHash) {
Index: clang/lib/Frontend/CompilerInvocation.cpp
===================================================================
--- clang/lib/Frontend/CompilerInvocation.cpp
+++ clang/lib/Frontend/CompilerInvocation.cpp
@@ -2069,6 +2069,8 @@
!Args.hasArg(OPT_fmodules_disable_diagnostic_validation);
Opts.ImplicitModuleMaps = Args.hasArg(OPT_fimplicit_module_maps);
Opts.ModuleMapFileHomeIsCwd = Args.hasArg(OPT_fmodule_map_file_home_is_cwd);
+ Opts.EnablePrebuiltImplicitModules =
+ Args.hasArg(OPT_fprebuilt_implicit_modules);
Opts.ModuleCachePruneInterval =
getLastArgIntValue(Args, OPT_fmodules_prune_interval, 7 * 24 * 60 * 60);
Opts.ModuleCachePruneAfter =
Index: clang/lib/Frontend/CompilerInstance.cpp
===================================================================
--- clang/lib/Frontend/CompilerInstance.cpp
+++ clang/lib/Frontend/CompilerInstance.cpp
@@ -1673,6 +1673,9 @@
!HSOpts.PrebuiltModulePaths.empty())) {
ModuleFileName =
PP->getHeaderSearchInfo().getPrebuiltModuleFileName(ModuleName);
+ if (HSOpts.EnablePrebuiltImplicitModules && ModuleFileName.empty())
+ ModuleFileName =
+ PP->getHeaderSearchInfo().getPrebuiltImplicitModuleFileName(Module);
if (!ModuleFileName.empty())
Source = PrebuiltModulePath;
}
Index: clang/lib/Driver/ToolChains/Clang.cpp
===================================================================
--- clang/lib/Driver/ToolChains/Clang.cpp
+++ clang/lib/Driver/ToolChains/Clang.cpp
@@ -2777,6 +2777,8 @@
std::string("-fprebuilt-module-path=") + A->getValue()));
A->claim();
}
+ if (Args.hasFlag(options::OPT_fprebuilt_implicit_modules, false))
+ CmdArgs.push_back("-fprebuilt-implicit-modules");
}
// -fmodule-name specifies the module that is currently being built (or
Index: clang/include/clang/Lex/HeaderSearchOptions.h
===================================================================
--- clang/include/clang/Lex/HeaderSearchOptions.h
+++ clang/include/clang/Lex/HeaderSearchOptions.h
@@ -141,6 +141,10 @@
/// file.
unsigned ModuleMapFileHomeIsCwd : 1;
+ /// Also search for prebuilt implicit modules in the prebuilt module cache
+ /// path.
+ unsigned EnablePrebuiltImplicitModules : 1;
+
/// The interval (in seconds) between pruning operations.
///
/// This operation is expensive, because it requires Clang to walk through
@@ -205,8 +209,9 @@
HeaderSearchOptions(StringRef _Sysroot = "/")
: Sysroot(_Sysroot), ModuleFormat("raw"), DisableModuleHash(false),
ImplicitModuleMaps(false), ModuleMapFileHomeIsCwd(false),
- UseBuiltinIncludes(true), UseStandardSystemIncludes(true),
- UseStandardCXXIncludes(true), UseLibcxx(false), Verbose(false),
+ EnablePrebuiltImplicitModules(false), UseBuiltinIncludes(true),
+ UseStandardSystemIncludes(true), UseStandardCXXIncludes(true),
+ UseLibcxx(false), Verbose(false),
ModulesValidateOncePerBuildSession(false),
ModulesValidateSystemHeaders(false), UseDebugInfo(false),
ModulesValidateDiagnosticOptions(true), ModulesHashContent(false) {}
Index: clang/include/clang/Lex/HeaderSearch.h
===================================================================
--- clang/include/clang/Lex/HeaderSearch.h
+++ clang/include/clang/Lex/HeaderSearch.h
@@ -505,6 +505,15 @@
std::string getPrebuiltModuleFileName(StringRef ModuleName,
bool FileMapOnly = false);
+ /// Retrieve the name of the prebuilt module file that should be used
+ /// to load the given module.
+ ///
+ /// \param Module The module whose module file name will be returned.
+ ///
+ /// \returns The name of the module file that corresponds to this module,
+ /// or an empty string if this module does not correspond to any module file.
+ std::string getPrebuiltImplicitModuleFileName(Module *Module);
+
/// Retrieve the name of the (to-be-)cached module file that should
/// be used to load a module with the given name.
///
@@ -601,6 +610,22 @@
Module *lookupModule(StringRef ModuleName, StringRef SearchName,
bool AllowExtraModuleMapSearch = false);
+ /// Retrieve the name of the (to-be-)cached module file that should
+ /// be used to load a module with the given name.
+ ///
+ /// \param ModuleName The module whose module file name will be returned.
+ ///
+ /// \param ModuleMapPath A path that when combined with \c ModuleName
+ /// uniquely identifies this module. See Module::ModuleMap.
+ ///
+ /// \param CachePath A path to the module cache.
+ ///
+ /// \returns The name of the module file that corresponds to this module,
+ /// or an empty string if this module does not correspond to any module file.
+ std::string getCachedModuleFileNameImpl(StringRef ModuleName,
+ StringRef ModuleMapPath,
+ StringRef CachePath);
+
/// Retrieve a module with the given name, which may be part of the
/// given framework.
///
Index: clang/include/clang/Driver/Options.td
===================================================================
--- clang/include/clang/Driver/Options.td
+++ clang/include/clang/Driver/Options.td
@@ -1341,6 +1341,9 @@
def fprebuilt_module_path : Joined<["-"], "fprebuilt-module-path=">, Group<i_Group>,
Flags<[DriverOption, CC1Option]>, MetaVarName<"<directory>">,
HelpText<"Specify the prebuilt module path">;
+def fprebuilt_implicit_modules : Flag <["-"], "fprebuilt-implicit-modules">, Group<f_Group>,
+ Flags<[DriverOption, CC1Option]>,
+ HelpText<"Look up implicit modules in the prebuilt module path">;
def fmodules_prune_interval : Joined<["-"], "fmodules-prune-interval=">, Group<i_Group>,
Flags<[CC1Option]>, MetaVarName<"<seconds>">,
HelpText<"Specify the interval (in seconds) between attempts to prune the module cache">;
Index: clang/docs/Modules.rst
===================================================================
--- clang/docs/Modules.rst
+++ clang/docs/Modules.rst
@@ -225,6 +225,100 @@
``-fprebuilt-module-path=<directory>``
Specify the path to the prebuilt modules. If specified, we will look for modules in this directory for a given top-level module name. We don't need a module map for loading prebuilt modules in this directory and the compiler will not try to rebuild these modules. This can be specified multiple times.
+``-fprebuilt-implicit-modules``
+ Enable prebuilt implicit modules. If a prebuilt module is not found in the
+ prebuilt modules paths (specified via ``-fprebuilt-module-path``), we will
+ look for a matching implicit modules in the prebuilt modules paths.
+
+Using Prebuilt Modules
+----------------------
+
+Below are a few examples illustrating uses of prebuilt modules via the different options.
+
+First, let's set up files for our examples.
+
+.. code-block:: c
+
+ /* A.h */
+ // empty
+
+.. code-block:: c
+
+ /* B.h */
+ #include "A.h"
+
+.. code-block:: c
+
+ /* use.c */
+ #include "B.h"
+
+.. code-block:: c
+
+ /* module.modulemap */
+ module A {
+ header "A.h"
+ }
+ module B {
+ header "B.h"
+ }
+
+In examples below, the compilation of ``use.c`` can be done without ``-cc1``, but the commands used to prebuild the modules would need to be updated to take into account the default options passed to ``clang -cc1``. (See ``clang use.c -v``)
+
+First let us use an explicit mapping from modules to files.
+
+.. code-block:: sh
+
+ rm -rf prebuilt ; mkdir prebuilt
+ clang -cc1 -x c module.modulemap -fmodules -emit-module -fmodule-name=A -o prebuilt/A.pcm
+ clang -cc1 -x c module.modulemap -fmodules -emit-module -fmodule-name=B -o prebuilt/B.pcm -fmodule-file=A=prebuilt/A.pcm
+ clang -cc1 -x c use.c -fmodules -o use.o -fmodule-file=A=prebuilt/A.pcm -fmodule-file=B=prebuilt/B.pcm
+
+Instead of of specifying the mappings manually, it can be convenient to use the ``-fprebuilt-module-path`` option.
+
+.. code-block:: sh
+
+ rm -rf prebuilt ; mkdir prebuilt
+ clang -cc1 -x c module.modulemap -fmodules -emit-module -fmodule-name=A -o prebuilt/A.pcm
+ clang -cc1 -x c module.modulemap -fmodules -emit-module -fmodule-name=B -o prebuilt/B.pcm -fmodule-file=A=prebuilt/A.pcm
+ clang -cc1 -x c use.c -fmodules -o use.o -fprebuilt-module-path=prebuilt
+
+A trick to prebuilt required modules in one command is to generate implicit modules using the ``-fdisable-module-hash`` option.
+
+.. code-block:: sh
+
+ rm -rf prebuilt ; mkdir prebuilt
+ clang -cc1 -x c module.modulemap -fmodules -emit-module -fmodule-name=B -fmodules-cache-path=prebuilt -fdisable-module-hash
+ ls prebuilt/*.pcm
+ # prebuilt/A.pcm prebuilt/B.pcm
+ clang -cc1 -x c use.c -fmodules -fprebuilt-module-path=prebuilt -o /use.o
+
+Maintaining multiple versions of prebuilt modules requires using a different mapping or pointing to a different prebuilt modules cache path. For example:
+
+.. code-block:: sh
+
+ rm -rf prebuilt_1 ; rm -rf prebuilt_2 ; mkdir prebuilt_1 ; mkdir prebuilt_2
+ clang -cc1 -x c module.modulemap -fmodules -emit-module -fmodule-name=B -fmodules-cache-path=prebuilt_1 -D_CONFIG_1
+ clang -cc1 -x c module.modulemap -fmodules -emit-module -fmodule-name=B -fmodules-cache-path=prebuilt_2 -D_CONFIG_2
+ clang -cc1 -x c use.c -o use.o -fmodules -fprebuilt-module-path=prebuilt_1 -DCONFIG_1
+ clang -cc1 -x c use.c -o use.o -fmodules -fprebuilt-module-path=prebuilt_2 -DCONFIG_2
+
+Instead, one can build implicit modules in a given cache path (using
+``-fmodules-cache-path``, and reuse them as prebuilt implicit modules by
+passing ``-fprebuilt-module-path`` and ``-fprebuilt-implicit-modules``.
+
+.. code-block:: sh
+
+ rm -rf prebuilt; mkdir prebuilt
+ clang -cc1 -x c module.modulemap -fmodules -emit-module -fmodule-name=B -fmodules-cache-path=prebuilt -D_CONFIG_1
+ clang -cc1 -x c module.modulemap -fmodules -emit-module -fmodule-name=B -fmodules-cache-path=prebuilt -D_CONFIG_2
+ find prebuilt -name "*.pcm"
+ # prebuilt/W7OPMD3I127H/A-285BFOGQN82QX.pcm
+ # prebuilt/W7OPMD3I127H/B-285BFOGQN82QX.pcm
+ # prebuilt/Y4JCOK22SBQC/A-285BFOGQN82QX.pcm
+ # prebuilt/Y4JCOK22SBQC/B-285BFOGQN82QX.pcm
+ clang -cc1 -x c use.c -o /use.o -fmodules -fmodules-cache-path=cache -fprebuilt-module-path=prebuilt -fprebuilt-implicit-modules -D_CONFIG_1
+ clang -cc1 -x c use.c -o /use.o -fmodules -fmodules-cache-path=cache -fprebuilt-module-path=prebuilt -fprebuilt-implicit-modules -D_CONFIG_2
+
Module Semantics
================
_______________________________________________
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
