[PATCH] D68997 [Modules] Allow searching for prebuilt implicit modules

[Alexandre Rames via cfe-commits]

Hello,

This is a proposal to allow searching for "implicit modules" in the prebuilt 
module cache paths.

https://reviews.llvm.org/D68997 

The current command-line options for prebuilt modules do not allow to easily
maintain and use multiple versions of modules. Both the producer and users of
prebuilt modules are required to know the relationships between compilation
options and module file paths. Using a particular version of a prebuilt module
requires passing a particular option on the command line (e.g.
-fmodule-file=[=] or -fprebuilt-module-path=).

However the compiler already knows how to distinguish and automatically locate
implicit modules. Hence this proposal to introduce the
-fprebuilt-implicit-modules option. When set, it enables searching for
implicit modules in the prebuilt module paths (specified via
-fprebuilt-module-path). To not modify existing behavior, this search takes
place after the standard search for prebuilt modules. If not

Here is a workflow illustrating how both the producer and consumer of prebuilt
modules would need to know what versions of prebuilt modules are available and
where they are located.

clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules_v1 
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules_v2 
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules_v3 

clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap 
-fprebuilt-module-path=prebuilt_modules_v1 
clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap 
With prebuilt implicit modules, the producer can generate prebuilt modules as
usual, all in the same output directory. The same mechanisms as for implicit
modules take care of incorporating hashes in the path to distinguish between
module versions.

Note that we do not specify the output module filename, so -o implicit modules 
are generated in the cache path prebuilt_modules.

clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules 
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules 
clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules 
The user can now simply enable prebuilt implicit modules and point to the
prebuilt modules cache. No need to "parse" command-line options to decide
what prebuilt modules (paths) to use.

clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap 
-fprebuilt-module-path=prebuilt_modules -fprebuilt-implicit-modules 
clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap 
-fprebuilt-module-path=prebuilt_modules -fprebuilt-implicit-modules 

This is for example particularly useful in a use-case where compilation is
expensive, and the configurations expected to be used are predictable, but not
controlled by the producer of prebuilt modules. Modules for the set of
predictable configurations can be prebuilt, and using them does not require
"parsing" the configuration (command-line options).

Regards,

Alexandre___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] 71e108c - Allow searching for prebuilt implicit modules.

[Alexandre Rames via cfe-commits]

Author: Alexandre Rames
Date: 2020-11-05T13:10:53-08:00
New Revision: 71e108cd86e70b06c5fa3a63689dcb3555c3d13f

URL: 
https://github.com/llvm/llvm-project/commit/71e108cd86e70b06c5fa3a63689dcb3555c3d13f
DIFF: 
https://github.com/llvm/llvm-project/commit/71e108cd86e70b06c5fa3a63689dcb3555c3d13f.diff

LOG: Allow searching for prebuilt implicit modules.

The behavior is controlled by the `-fprebuilt-implicit-modules` option, and
allows searching for implicit modules in the prebuilt module cache paths.

The current command-line options for prebuilt modules do not allow to easily
maintain and use multiple versions of modules. Both the producer and users of
prebuilt modules are required to know the relationships between compilation
options and module file paths. Using a particular version of a prebuilt module
requires passing a particular option on the command line (e.g.
`-fmodule-file=[=]` or `-fprebuilt-module-path=`).

However the compiler already knows how to distinguish and automatically locate
implicit modules. Hence this proposal to introduce the
`-fprebuilt-implicit-modules` option. When set, it enables searching for
implicit modules in the prebuilt module paths (specified via
`-fprebuilt-module-path`). To not modify existing behavior, this search takes
place after the standard search for prebuilt modules. If not

Here is a workflow illustrating how both the producer and consumer of prebuilt
modules would need to know what versions of prebuilt modules are available and
where they are located.

  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules_v1 
  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules_v2 
  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules_v3 

  clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap 
-fprebuilt-module-path=prebuilt_modules_v1 
  clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap 

With prebuilt implicit modules, the producer can generate prebuilt modules as
usual, all in the same output directory. The same mechanisms as for implicit
modules take care of incorporating hashes in the path to distinguish between
module versions.

Note that we do not specify the output module filename, so `-o` implicit 
modules are generated in the cache path `prebuilt_modules`.

  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules 
  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules 
  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo 
-fmodules-cache-path=prebuilt_modules 

The user can now simply enable prebuilt implicit modules and point to the
prebuilt modules cache. No need to "parse" command-line options to decide
what prebuilt modules (paths) to use.

  clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap 
-fprebuilt-module-path=prebuilt_modules -fprebuilt-implicit-modules 
  clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap 
-fprebuilt-module-path=prebuilt_modules -fprebuilt-implicit-modules 


This is for example particularly useful in a use-case where compilation is
expensive, and the configurations expected to be used are predictable, but not
controlled by the producer of prebuilt modules. Modules for the set of
predictable configurations can be prebuilt, and using them does not require
"parsing" the configuration (command-line options).

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D68997

Added: 
clang/test/Modules/Inputs/prebuilt-implicit-module/a.h
clang/test/Modules/Inputs/prebuilt-implicit-module/module.modulemap
clang/test/Modules/prebuilt-implicit-modules.m

Modified: 
clang/docs/Modules.rst
clang/include/clang/Driver/Options.td
clang/include/clang/Frontend/CompilerInstance.h
clang/include/clang/Lex/HeaderSearch.h
clang/include/clang/Lex/HeaderSearchOptions.h
clang/lib/Driver/ToolChains/Clang.cpp
clang/lib/Frontend/CompilerInstance.cpp
clang/lib/Frontend/CompilerInvocation.cpp
clang/lib/Lex/HeaderSearch.cpp
clang/lib/Serialization/ASTReader.cpp
clang/lib/Serialization/ASTWriter.cpp

Removed: 




diff  --git a/clang/docs/Modules.rst b/clang/docs/Modules.rst
index 63f09f90fe6c..703ba86c68a0 100644
--- a/clang/docs/Modules.rst
+++ b/clang/docs/Modules.rst
@@ -225,6 +225,11 @@ Command-line parameters
 ``-fprebuilt-module-path=``
   Specify the path to the prebuilt modules. If specified, we will look for 
modules in this directory for a given top-level module name. We don't need a 
module map for loading prebuilt modules in this directory and the compiler will 
not try to rebuild these modules. This can be specified multiple times.
 
+``-fprebuilt-implicit-modules``
+  Enable prebuilt implicit mo

[clang] 58c586e - Allow searching for prebuilt implicit modules.

[Alexandre Rames via cfe-commits]

Author: Alexandre Rames
Date: 2020-11-10T10:14:13-08:00
New Revision: 58c586e701889250c08285193dc75d94a7ed302d

URL: 
https://github.com/llvm/llvm-project/commit/58c586e701889250c08285193dc75d94a7ed302d
DIFF: 
https://github.com/llvm/llvm-project/commit/58c586e701889250c08285193dc75d94a7ed302d.diff

LOG: Allow searching for prebuilt implicit modules.

This reverts commit c67656b994c87224e0b33e2c4b09093986a5cfa6, and addresses the
build issue.

Added: 
clang/test/Modules/Inputs/prebuilt-implicit-module/a.h
clang/test/Modules/Inputs/prebuilt-implicit-module/module.modulemap
clang/test/Modules/prebuilt-implicit-modules.m

Modified: 
clang/docs/Modules.rst
clang/include/clang/Driver/Options.td
clang/include/clang/Frontend/CompilerInstance.h
clang/include/clang/Lex/HeaderSearch.h
clang/include/clang/Lex/HeaderSearchOptions.h
clang/lib/Driver/ToolChains/Clang.cpp
clang/lib/Frontend/CompilerInstance.cpp
clang/lib/Frontend/CompilerInvocation.cpp
clang/lib/Lex/HeaderSearch.cpp
clang/lib/Serialization/ASTReader.cpp
clang/lib/Serialization/ASTWriter.cpp

Removed: 




diff  --git a/clang/docs/Modules.rst b/clang/docs/Modules.rst
index 63f09f90fe6c..703ba86c68a0 100644
--- a/clang/docs/Modules.rst
+++ b/clang/docs/Modules.rst
@@ -225,6 +225,11 @@ Command-line parameters
 ``-fprebuilt-module-path=``
   Specify the path to the prebuilt modules. If specified, we will look for 
modules in this directory for a given top-level module name. We don't need a 
module map for loading prebuilt modules in this directory and the compiler will 
not try to rebuild these modules. This can be specified multiple times.
 
+``-fprebuilt-implicit-modules``
+  Enable prebuilt implicit modules. If a prebuilt module is not found in the
+  prebuilt modules paths (specified via ``-fprebuilt-module-path``), we will
+  look for a matching implicit module in the prebuilt modules paths.
+
 -cc1 Options
 
 
@@ -235,6 +240,123 @@ Command-line parameters
   being built if the command line arguments are not homogeneous across your
   build.
 
+Using Prebuilt Modules
+--
+
+Below are a few examples illustrating uses of prebuilt modules via the 
diff erent options.
+
+First, let's set up files for our examples.
+
+.. code-block:: c
+
+  /* A.h */
+  #ifdef ENABLE_A
+  void a() {}
+  #endif
+
+.. code-block:: c
+
+  /* B.h */
+  #include "A.h"
+
+.. code-block:: c
+
+  /* use.c */
+  #include "B.h"
+  void use() {
+  #ifdef ENABLE_A
+a();
+  #endif
+  }
+
+.. code-block:: c
+
+  /* module.modulemap */
+  module A {
+header "A.h"
+  }
+  module B {
+header "B.h"
+export *
+  }
+
+In the examples below, the compilation of ``use.c`` can be done without 
``-cc1``, but the commands used to prebuild the modules would need to be 
updated to take into account the default options passed to ``clang -cc1``. (See 
``clang use.c -v``)
+Note also that, since we use ``-cc1``, we specify the ``-fmodule-map-file=`` 
or ``-fimplicit-module-maps`` options explicitly. When using the clang driver, 
``-fimplicit-module-maps`` is implied by ``-fmodules``.
+
+First let us use an explicit mapping from modules to files.
+
+.. code-block:: sh
+
+  rm -rf prebuilt ; mkdir prebuilt
+  clang -cc1 -emit-module -o prebuilt/A.pcm -fmodules module.modulemap 
-fmodule-name=A
+  clang -cc1 -emit-module -o prebuilt/B.pcm -fmodules module.modulemap 
-fmodule-name=B -fmodule-file=A=prebuilt/A.pcm
+  clang -cc1 -emit-obj use.c -fmodules -fmodule-map-file=module.modulemap 
-fmodule-file=A=prebuilt/A.pcm -fmodule-file=B=prebuilt/B.pcm
+
+Instead of of specifying the mappings manually, it can be convenient to use 
the ``-fprebuilt-module-path`` option. Let's also use 
``-fimplicit-module-maps`` instead of manually pointing to our module map.
+
+.. code-block:: sh
+
+  rm -rf prebuilt; mkdir prebuilt
+  clang -cc1 -emit-module -o prebuilt/A.pcm -fmodules module.modulemap 
-fmodule-name=A
+  clang -cc1 -emit-module -o prebuilt/B.pcm -fmodules module.modulemap 
-fmodule-name=B -fprebuilt-module-path=prebuilt
+  clang -cc1 -emit-obj use.c -fmodules -fimplicit-module-maps 
-fprebuilt-module-path=prebuilt
+
+A trick to prebuild all modules required for our source file in one command is 
to generate implicit modules while using the ``-fdisable-module-hash`` option.
+
+.. code-block:: sh
+
+  rm -rf prebuilt ; mkdir prebuilt
+  clang -cc1 -emit-obj use.c -fmodules -fimplicit-module-maps 
-fmodules-cache-path=prebuilt -fdisable-module-hash
+  ls prebuilt/*.pcm
+  # prebuilt/A.pcm  prebuilt/B.pcm
+
+Note that with explicit or prebuilt modules, we are responsible for, and 
should be particularly careful about the compatibility of our modules.
+Using mismatching compilation options and modules may lead to issues.
+
+.. code-block:: sh
+
+  clang -cc1 -emit-obj use.c -fmodules -fimplicit-module-maps 
-fprebuilt-module-path=p

[clang] 655bea4 - [modules] Use `HashBuilder` and `MD5` for the module hash.

[Alexandre Rames via cfe-commits]

Author: Alexandre Rames
Date: 2021-09-03T11:13:36-07:00
New Revision: 655bea4226b401a11164f99c6344e38d8742b8e4

URL: 
https://github.com/llvm/llvm-project/commit/655bea4226b401a11164f99c6344e38d8742b8e4
DIFF: 
https://github.com/llvm/llvm-project/commit/655bea4226b401a11164f99c6344e38d8742b8e4.diff

LOG: [modules] Use `HashBuilder` and `MD5` for the module hash.

Per the comments, `hash_code` values "are not stable to save or
persist", so are unsuitable for the module hash, which must persist
across compilations for the implicit module hashes to match. Note that
in practice, today, `hash_code` are stable. But this is an
implementation detail, with a clear `FIXME` indicating we should switch
to a per-execution seed.

The stability of `MD5` also allows modules cross-compilation use-cases.
The `size_t` underlying storage for `hash_code` varying across platforms
could cause mismatching hashes when cross-compiling from a 64bit
target to a 32bit target.

Note that native endianness is still used for the hash computation. So hashes
will differ between platforms of different endianness.

Reviewed By: jansvoboda11

Differential Revision: https://reviews.llvm.org/D102943

Added: 


Modified: 
clang/include/clang/Basic/ObjCRuntime.h
clang/include/clang/Basic/Sanitizers.h
clang/include/clang/Lex/HeaderSearchOptions.h
clang/include/clang/Serialization/ModuleFileExtension.h
clang/lib/Frontend/CompilerInvocation.cpp
clang/lib/Frontend/TestModuleFileExtension.cpp
clang/lib/Frontend/TestModuleFileExtension.h
clang/lib/Serialization/ModuleFileExtension.cpp
clang/unittests/Frontend/CompilerInvocationTest.cpp
llvm/include/llvm/Support/VersionTuple.h

Removed: 




diff  --git a/clang/include/clang/Basic/ObjCRuntime.h 
b/clang/include/clang/Basic/ObjCRuntime.h
index 26403bfa98c9d..30a5fde407541 100644
--- a/clang/include/clang/Basic/ObjCRuntime.h
+++ b/clang/include/clang/Basic/ObjCRuntime.h
@@ -18,6 +18,7 @@
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/HashBuilder.h"
 #include "llvm/Support/VersionTuple.h"
 #include 
 
@@ -480,6 +481,12 @@ class ObjCRuntime {
   friend llvm::hash_code hash_value(const ObjCRuntime &OCR) {
 return llvm::hash_combine(OCR.getKind(), OCR.getVersion());
   }
+
+  template 
+  friend void addHash(llvm::HashBuilderImpl &HBuilder,
+  const ObjCRuntime &OCR) {
+HBuilder.add(OCR.getKind(), OCR.getVersion());
+  }
 };
 
 raw_ostream &operator<<(raw_ostream &out, const ObjCRuntime &value);

diff  --git a/clang/include/clang/Basic/Sanitizers.h 
b/clang/include/clang/Basic/Sanitizers.h
index b12a3b7821d7a..db53010645ae3 100644
--- a/clang/include/clang/Basic/Sanitizers.h
+++ b/clang/include/clang/Basic/Sanitizers.h
@@ -16,6 +16,7 @@
 
 #include "clang/Basic/LLVM.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/HashBuilder.h"
 #include "llvm/Transforms/Instrumentation/AddressSanitizerOptions.h"
 #include 
 #include 
@@ -72,6 +73,12 @@ class SanitizerMask {
 
   llvm::hash_code hash_value() const;
 
+  template 
+  friend void addHash(llvm::HashBuilderImpl &HBuilder,
+  const SanitizerMask &SM) {
+HBuilder.addRange(&SM.maskLoToHigh[0], &SM.maskLoToHigh[kNumElem]);
+  }
+
   constexpr explicit operator bool() const {
 return maskLoToHigh[0] || maskLoToHigh[1];
   }

diff  --git a/clang/include/clang/Lex/HeaderSearchOptions.h 
b/clang/include/clang/Lex/HeaderSearchOptions.h
index 42f3cff8c57ae..4efdfc26c3c67 100644
--- a/clang/include/clang/Lex/HeaderSearchOptions.h
+++ b/clang/include/clang/Lex/HeaderSearchOptions.h
@@ -14,10 +14,11 @@
 #include "llvm/ADT/Hashing.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/HashBuilder.h"
 #include 
+#include 
 #include 
 #include 
-#include 
 
 namespace clang {
 
@@ -256,11 +257,23 @@ inline llvm::hash_code hash_value(const 
HeaderSearchOptions::Entry &E) {
   return llvm::hash_combine(E.Path, E.Group, E.IsFramework, E.IgnoreSysRoot);
 }
 
+template 
+inline void addHash(llvm::HashBuilderImpl &HBuilder,
+const HeaderSearchOptions::Entry &E) {
+  HBuilder.add(E.Path, E.Group, E.IsFramework, E.IgnoreSysRoot);
+}
+
 inline llvm::hash_code
 hash_value(const HeaderSearchOptions::SystemHeaderPrefix &SHP) {
   return llvm::hash_combine(SHP.Prefix, SHP.IsSystemHeader);
 }
 
+template 
+inline void addHash(llvm::HashBuilderImpl &HBuilder,
+const HeaderSearchOptions::SystemHeaderPrefix &SHP) {
+  HBuilder.add(SHP.Prefix, SHP.IsSystemHeader);
+}
+
 } // namespace clang
 
 #endif // LLVM_CLANG_LEX_HEADERSEARCHOPTIONS_H

diff  --git a/clang/include/clang/Serialization/ModuleFileExtension.h 
b/clang/include/clang/Serialization/ModuleFileExtension.h
index 34ea870724a4c..3e84a65c4b805 100644
--- a/clang/include/clang/Serialization