Thanks Eric! On Mon, Oct 14, 2019 at 8:12 PM Eric Christopher <echri...@gmail.com> wrote: > > This was breaking a few bots and I couldn't find you on irc so I've > reverted it thusly: > > echristo@jhereg ~/s/llvm-project> git llvm push > Pushing 1 commit: > 175b1b856ea Temporarily Revert [Modules][PCH] Hash input files > content as it's breaking a few bots. > Sending cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td > Sending cfe/trunk/include/clang/Driver/Options.td > Sending cfe/trunk/include/clang/Lex/HeaderSearchOptions.h > Sending cfe/trunk/include/clang/Serialization/ASTBitCodes.h > Sending cfe/trunk/include/clang/Serialization/ASTReader.h > Sending cfe/trunk/lib/Driver/ToolChains/Clang.cpp > Sending cfe/trunk/lib/Frontend/CompilerInstance.cpp > Sending cfe/trunk/lib/Frontend/CompilerInvocation.cpp > Sending cfe/trunk/lib/Serialization/ASTReader.cpp > Sending cfe/trunk/lib/Serialization/ASTWriter.cpp > Deleting cfe/trunk/test/Modules/validate-file-content.m > Deleting cfe/trunk/test/PCH/validate-file-content.m > Transmitting file data ..........done > Committing transaction... > Committed revision 374842. > Committed 175b1b856ea to svn. > > Sorry for the inconvenience! > > -eric > > On Mon, Oct 14, 2019 at 3:59 PM Bruno Cardoso Lopes via cfe-commits > <cfe-commits@lists.llvm.org> wrote: > > > > Author: bruno > > Date: Mon Oct 14 16:02:03 2019 > > New Revision: 374841 > > > > URL: http://llvm.org/viewvc/llvm-project?rev=374841&view=rev > > Log: > > [Modules][PCH] Hash input files content > > > > Summary: > > When files often get touched during builds, the mtime based validation > > leads to different problems in implicit modules builds, even when the > > content doesn't actually change: > > > > - Modules only: module invalidation due to out of date files. Usually > > causing rebuild traffic. > > - Modules + PCH: build failures because clang cannot rebuild a module if it > > comes from building a PCH. > > - PCH: build failures because clang cannot rebuild a PCH in case one of the > > input headers has different mtime. > > > > This patch proposes hashing the content of input files (headers and > > module maps), which is performed during serialization time. When looking > > at input files for validation, clang only computes the hash in case > > there's a mtime mismatch. > > > > I've tested a couple of different hash algorithms availble in LLVM in > > face of building modules+pch for `#import <Cocoa/Cocoa.h>`: > > - `hash_code`: performace diff within the noise, total module cache > > increased by 0.07%. > > - `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be > > BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from > > `hash_code`. > > - `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file. > > > > Given the numbers above, the patch uses `hash_code`. The patch also > > improves invalidation error msgs to point out which type of problem the > > user is facing: "mtime", "size" or "content". > > > > rdar://problem/29320105 > > > > Reviewers: dexonsmith, arphaman, rsmith, aprantl > > > > Subscribers: jkorous, cfe-commits, ributzka > > > > Tags: #clang > > > > Differential Revision: https://reviews.llvm.org/D67249 > > > > Added: > > cfe/trunk/test/Modules/validate-file-content.m > > cfe/trunk/test/PCH/validate-file-content.m > > Modified: > > cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td > > cfe/trunk/include/clang/Driver/Options.td > > cfe/trunk/include/clang/Lex/HeaderSearchOptions.h > > cfe/trunk/include/clang/Serialization/ASTBitCodes.h > > cfe/trunk/include/clang/Serialization/ASTReader.h > > cfe/trunk/lib/Driver/ToolChains/Clang.cpp > > cfe/trunk/lib/Frontend/CompilerInstance.cpp > > cfe/trunk/lib/Frontend/CompilerInvocation.cpp > > cfe/trunk/lib/Serialization/ASTReader.cpp > > cfe/trunk/lib/Serialization/ASTWriter.cpp > > > > Modified: cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td?rev=374841&r1=374840&r2=374841&view=diff > > ============================================================================== > > --- cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td (original) > > +++ cfe/trunk/include/clang/Basic/DiagnosticSerializationKinds.td Mon Oct > > 14 16:02:03 2019 > > @@ -18,13 +18,16 @@ def err_fe_pch_malformed : Error< > > def err_fe_pch_malformed_block : Error< > > "malformed block record in PCH file: '%0'">, DefaultFatal; > > def err_fe_pch_file_modified : Error< > > - "file '%0' has been modified since the precompiled header '%1' was > > built">, > > + "file '%0' has been modified since the precompiled header '%1' was > > built" > > + ": %select{size|mtime|content}2 changed">, > > DefaultFatal; > > def err_fe_module_file_modified : Error< > > - "file '%0' has been modified since the module file '%1' was built">, > > + "file '%0' has been modified since the module file '%1' was built" > > + ": %select{size|mtime|content}2 changed">, > > DefaultFatal; > > def err_fe_ast_file_modified : Error< > > - "file '%0' has been modified since the AST file '%1' was built">, > > + "file '%0' has been modified since the AST file '%1' was built" > > + ": %select{size|mtime|content}2 changed">, > > DefaultFatal; > > def err_fe_pch_file_overridden : Error< > > "file '%0' from the precompiled header has been overridden">; > > @@ -399,6 +402,8 @@ def warn_module_uses_date_time : Warning > > def err_module_no_size_mtime_for_header : Error< > > "cannot emit module %0: %select{size|mtime}1 must be explicitly > > specified " > > "for missing header file \"%2\"">; > > +def err_module_unable_to_hash_content : Error< > > + "failed to hash content for '%0' because memory buffer cannot be > > retrieved">; > > } // let CategoryName > > } // let Component > > > > > > Modified: cfe/trunk/include/clang/Driver/Options.td > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Driver/Options.td?rev=374841&r1=374840&r2=374841&view=diff > > ============================================================================== > > --- cfe/trunk/include/clang/Driver/Options.td (original) > > +++ cfe/trunk/include/clang/Driver/Options.td Mon Oct 14 16:02:03 2019 > > @@ -1368,6 +1368,28 @@ def fmodules_validate_system_headers : F > > HelpText<"Validate the system headers that a module depends on when > > loading the module">; > > def fno_modules_validate_system_headers : Flag<["-"], > > "fno-modules-validate-system-headers">, > > Group<i_Group>, Flags<[DriverOption]>; > > + > > +def fvalidate_ast_input_files_content: > > + Flag <["-"], "fvalidate-ast-input-files-content">, > > + Group<f_Group>, Flags<[CC1Option]>, > > + HelpText<"Compute and store the hash of input files used to build an > > AST." > > + " Files with mismatching mtime's are considered valid" > > + " if both contents is identical">; > > +def fmodules_validate_input_files_content: > > + Flag <["-"], "fmodules-validate-input-files-content">, > > + Group<f_Group>, Flags<[DriverOption]>, > > + HelpText<"Validate PCM input files based on content if mtime differs">; > > +def fno_modules_validate_input_files_content: > > + Flag <["-"], "fno_modules-validate-input-files-content">, > > + Group<f_Group>, Flags<[DriverOption]>; > > +def fpch_validate_input_files_content: > > + Flag <["-"], "fpch-validate-input-files-content">, > > + Group<f_Group>, Flags<[DriverOption]>, > > + HelpText<"Validate PCH input files based on content if mtime differs">; > > +def fno_pch_validate_input_files_content: > > + Flag <["-"], "fno_pch-validate-input-files-content">, > > + Group<f_Group>, Flags<[DriverOption]>; > > + > > def fmodules : Flag <["-"], "fmodules">, Group<f_Group>, > > Flags<[DriverOption, CC1Option]>, > > HelpText<"Enable the 'modules' language feature">; > > > > Modified: cfe/trunk/include/clang/Lex/HeaderSearchOptions.h > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Lex/HeaderSearchOptions.h?rev=374841&r1=374840&r2=374841&view=diff > > ============================================================================== > > --- cfe/trunk/include/clang/Lex/HeaderSearchOptions.h (original) > > +++ cfe/trunk/include/clang/Lex/HeaderSearchOptions.h Mon Oct 14 16:02:03 > > 2019 > > @@ -195,6 +195,10 @@ public: > > /// Whether to validate system input files when a module is loaded. > > unsigned ModulesValidateSystemHeaders : 1; > > > > + // Whether the content of input files should be hashed and used to > > + // validate consistency. > > + unsigned ValidateASTInputFilesContent : 1; > > + > > /// Whether the module includes debug information (-gmodules). > > unsigned UseDebugInfo : 1; > > > > @@ -208,7 +212,8 @@ public: > > UseBuiltinIncludes(true), UseStandardSystemIncludes(true), > > UseStandardCXXIncludes(true), UseLibcxx(false), Verbose(false), > > ModulesValidateOncePerBuildSession(false), > > - ModulesValidateSystemHeaders(false), UseDebugInfo(false), > > + ModulesValidateSystemHeaders(false), > > + ValidateASTInputFilesContent(false), UseDebugInfo(false), > > ModulesValidateDiagnosticOptions(true), ModulesHashContent(false) > > {} > > > > /// AddPath - Add the \p Path path to the specified \p Group list. > > > > Modified: cfe/trunk/include/clang/Serialization/ASTBitCodes.h > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTBitCodes.h?rev=374841&r1=374840&r2=374841&view=diff > > ============================================================================== > > --- cfe/trunk/include/clang/Serialization/ASTBitCodes.h (original) > > +++ cfe/trunk/include/clang/Serialization/ASTBitCodes.h Mon Oct 14 16:02:03 > > 2019 > > @@ -382,7 +382,10 @@ namespace serialization { > > /// inside the control block. > > enum InputFileRecordTypes { > > /// An input file. > > - INPUT_FILE = 1 > > + INPUT_FILE = 1, > > + > > + /// The input file content hash > > + INPUT_FILE_HASH > > }; > > > > /// Record types that occur within the AST block itself. > > > > Modified: cfe/trunk/include/clang/Serialization/ASTReader.h > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Serialization/ASTReader.h?rev=374841&r1=374840&r2=374841&view=diff > > ============================================================================== > > --- cfe/trunk/include/clang/Serialization/ASTReader.h (original) > > +++ cfe/trunk/include/clang/Serialization/ASTReader.h Mon Oct 14 16:02:03 > > 2019 > > @@ -930,6 +930,9 @@ private: > > /// Whether validate system input files. > > bool ValidateSystemInputs; > > > > + /// Whether validate headers and module maps using hash based on > > contents. > > + bool ValidateASTInputFilesContent; > > + > > /// Whether we are allowed to use the global module index. > > bool UseGlobalIndex; > > > > @@ -1203,6 +1206,7 @@ private: > > > > struct InputFileInfo { > > std::string Filename; > > + uint64_t ContentHash; > > off_t StoredSize; > > time_t StoredTime; > > bool Overridden; > > @@ -1437,6 +1441,8 @@ private: > > void Error(StringRef Msg) const; > > void Error(unsigned DiagID, StringRef Arg1 = StringRef(), > > StringRef Arg2 = StringRef()) const; > > + void Error(unsigned DiagID, StringRef Arg1, StringRef Arg2, > > + unsigned Select) const; > > void Error(llvm::Error &&Err) const; > > > > public: > > @@ -1485,7 +1491,9 @@ public: > > StringRef isysroot = "", bool DisableValidation = false, > > bool AllowASTWithCompilerErrors = false, > > bool AllowConfigurationMismatch = false, > > - bool ValidateSystemInputs = false, bool UseGlobalIndex = true, > > + bool ValidateSystemInputs = false, > > + bool ValidateASTInputFilesContent = false, > > + bool UseGlobalIndex = true, > > std::unique_ptr<llvm::Timer> ReadTimer = {}); > > ASTReader(const ASTReader &) = delete; > > ASTReader &operator=(const ASTReader &) = delete; > > > > Modified: cfe/trunk/lib/Driver/ToolChains/Clang.cpp > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Driver/ToolChains/Clang.cpp?rev=374841&r1=374840&r2=374841&view=diff > > ============================================================================== > > --- cfe/trunk/lib/Driver/ToolChains/Clang.cpp (original) > > +++ cfe/trunk/lib/Driver/ToolChains/Clang.cpp Mon Oct 14 16:02:03 2019 > > @@ -2777,6 +2777,10 @@ static void RenderModulesOptions(Compila > > std::string("-fprebuilt-module-path=") + A->getValue())); > > A->claim(); > > } > > + if (Args.hasFlag(options::OPT_fmodules_validate_input_files_content, > > + options::OPT_fno_modules_validate_input_files_content, > > + false)) > > + CmdArgs.push_back("-fvalidate-ast-input-files-content"); > > } > > > > // -fmodule-name specifies the module that is currently being built (or > > @@ -4899,6 +4903,10 @@ void Clang::ConstructJob(Compilation &C, > > Std && (Std->containsValue("c++2a") || > > Std->containsValue("c++latest")); > > RenderModulesOptions(C, D, Args, Input, Output, CmdArgs, HaveModules); > > > > + if (Args.hasFlag(options::OPT_fpch_validate_input_files_content, > > + options::OPT_fno_pch_validate_input_files_content, > > false)) > > + CmdArgs.push_back("-fvalidate-ast-input-files-content"); > > + > > Args.AddLastArg(CmdArgs, options::OPT_fexperimental_new_pass_manager, > > options::OPT_fno_experimental_new_pass_manager); > > > > > > Modified: cfe/trunk/lib/Frontend/CompilerInstance.cpp > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Frontend/CompilerInstance.cpp?rev=374841&r1=374840&r2=374841&view=diff > > ============================================================================== > > --- cfe/trunk/lib/Frontend/CompilerInstance.cpp (original) > > +++ cfe/trunk/lib/Frontend/CompilerInstance.cpp Mon Oct 14 16:02:03 2019 > > @@ -510,7 +510,8 @@ IntrusiveRefCntPtr<ASTReader> CompilerIn > > PP, ModuleCache, &Context, PCHContainerRdr, Extensions, > > Sysroot.empty() ? "" : Sysroot.data(), DisablePCHValidation, > > AllowPCHWithCompilerErrors, /*AllowConfigurationMismatch*/ false, > > - HSOpts.ModulesValidateSystemHeaders, UseGlobalModuleIndex)); > > + HSOpts.ModulesValidateSystemHeaders, > > HSOpts.ValidateASTInputFilesContent, > > + UseGlobalModuleIndex)); > > > > // We need the external source to be set up before we read the AST, > > because > > // eagerly-deserialized declarations may use it. > > @@ -1492,6 +1493,7 @@ void CompilerInstance::createModuleManag > > /*AllowASTWithCompilerErrors=*/false, > > /*AllowConfigurationMismatch=*/false, > > HSOpts.ModulesValidateSystemHeaders, > > + HSOpts.ValidateASTInputFilesContent, > > getFrontendOpts().UseGlobalModuleIndex, std::move(ReadTimer)); > > if (hasASTConsumer()) { > > ModuleManager->setDeserializationListener( > > > > Modified: cfe/trunk/lib/Frontend/CompilerInvocation.cpp > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Frontend/CompilerInvocation.cpp?rev=374841&r1=374840&r2=374841&view=diff > > ============================================================================== > > --- cfe/trunk/lib/Frontend/CompilerInvocation.cpp (original) > > +++ cfe/trunk/lib/Frontend/CompilerInvocation.cpp Mon Oct 14 16:02:03 2019 > > @@ -2081,6 +2081,8 @@ static void ParseHeaderSearchArgs(Header > > getLastArgUInt64Value(Args, OPT_fbuild_session_timestamp, 0); > > Opts.ModulesValidateSystemHeaders = > > Args.hasArg(OPT_fmodules_validate_system_headers); > > + Opts.ValidateASTInputFilesContent = > > + Args.hasArg(OPT_fvalidate_ast_input_files_content); > > if (const Arg *A = Args.getLastArg(OPT_fmodule_format_EQ)) > > Opts.ModuleFormat = A->getValue(); > > > > > > Modified: cfe/trunk/lib/Serialization/ASTReader.cpp > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTReader.cpp?rev=374841&r1=374840&r2=374841&view=diff > > ============================================================================== > > --- cfe/trunk/lib/Serialization/ASTReader.cpp (original) > > +++ cfe/trunk/lib/Serialization/ASTReader.cpp Mon Oct 14 16:02:03 2019 > > @@ -1247,6 +1247,12 @@ void ASTReader::Error(unsigned DiagID, > > Diag(DiagID) << Arg1 << Arg2; > > } > > > > +void ASTReader::Error(unsigned DiagID, StringRef Arg1, StringRef Arg2, > > + unsigned Select) const { > > + if (!Diags.isDiagnosticInFlight()) > > + Diag(DiagID) << Arg1 << Arg2 << Select; > > +} > > + > > void ASTReader::Error(llvm::Error &&Err) const { > > Error(toString(std::move(Err))); > > } > > @@ -2241,6 +2247,24 @@ ASTReader::readInputFileInfo(ModuleFile > > R.TopLevelModuleMap = static_cast<bool>(Record[5]); > > R.Filename = Blob; > > ResolveImportedPath(F, R.Filename); > > + > > + Expected<llvm::BitstreamEntry> MaybeEntry = Cursor.advance(); > > + if (!MaybeEntry) // FIXME this drops errors on the floor. > > + consumeError(MaybeEntry.takeError()); > > + llvm::BitstreamEntry Entry = MaybeEntry.get(); > > + assert(Entry.Kind == llvm::BitstreamEntry::Record && > > + "expected record type for input file hash"); > > + > > + Record.clear(); > > + if (Expected<unsigned> Maybe = Cursor.readRecord(Entry.ID, Record)) > > + assert(static_cast<InputFileRecordTypes>(Maybe.get()) == > > INPUT_FILE_HASH && > > + "invalid record type for input file hash"); > > + else { > > + // FIXME this drops errors on the floor. > > + consumeError(Maybe.takeError()); > > + } > > + R.ContentHash = (static_cast<uint64_t>(Record[1]) << 32) | > > + static_cast<uint64_t>(Record[0]); > > return R; > > } > > > > @@ -2271,6 +2295,7 @@ InputFile ASTReader::getInputFile(Module > > bool Overridden = FI.Overridden; > > bool Transient = FI.Transient; > > StringRef Filename = FI.Filename; > > + uint64_t StoredContentHash = FI.ContentHash; > > > > const FileEntry *File = nullptr; > > if (auto FE = FileMgr.getFile(Filename, /*OpenFile=*/false)) > > @@ -2325,14 +2350,46 @@ InputFile ASTReader::getInputFile(Module > > } > > } > > > > - bool IsOutOfDate = false; > > + enum ModificationType { > > + Size, > > + ModTime, > > + Content, > > + None, > > + }; > > + auto HasInputFileChanged = [&]() { > > + if (StoredSize != File->getSize()) > > + return ModificationType::Size; > > + if (!DisableValidation && StoredTime && > > + StoredTime != File->getModificationTime()) { > > + // In case the modification time changes but not the content, > > + // accept the cached file as legit. > > + if (ValidateASTInputFilesContent && > > + StoredContentHash != static_cast<uint64_t>(llvm::hash_code(-1))) > > { > > + auto MemBuffOrError = FileMgr.getBufferForFile(File); > > + if (!MemBuffOrError) { > > + if (!Complain) > > + return ModificationType::ModTime; > > + std::string ErrorStr = "could not get buffer for file '"; > > + ErrorStr += File->getName(); > > + ErrorStr += "'"; > > + Error(ErrorStr); > > + return ModificationType::ModTime; > > + } > > + > > + auto ContentHash = hash_value(MemBuffOrError.get()->getBuffer()); > > + if (StoredContentHash == static_cast<uint64_t>(ContentHash)) > > + return ModificationType::None; > > + return ModificationType::Content; > > + } > > + return ModificationType::ModTime; > > + } > > + return ModificationType::None; > > + }; > > > > + bool IsOutOfDate = false; > > + auto FileChange = HasInputFileChanged(); > > // For an overridden file, there is nothing to validate. > > - if (!Overridden && // > > - (StoredSize != File->getSize() || > > - (StoredTime && StoredTime != File->getModificationTime() && > > - !DisableValidation) > > - )) { > > + if (!Overridden && FileChange != ModificationType::None) { > > if (Complain) { > > // Build a list of the PCH imports that got us here (in reverse). > > SmallVector<ModuleFile *, 4> ImportStack(1, &F); > > @@ -2341,13 +2398,17 @@ InputFile ASTReader::getInputFile(Module > > > > // The top-level PCH is stale. > > StringRef TopLevelPCHName(ImportStack.back()->FileName); > > - unsigned DiagnosticKind = > > moduleKindForDiagnostic(ImportStack.back()->Kind); > > + unsigned DiagnosticKind = > > + moduleKindForDiagnostic(ImportStack.back()->Kind); > > if (DiagnosticKind == 0) > > - Error(diag::err_fe_pch_file_modified, Filename, TopLevelPCHName); > > + Error(diag::err_fe_pch_file_modified, Filename, TopLevelPCHName, > > + (unsigned)FileChange); > > else if (DiagnosticKind == 1) > > - Error(diag::err_fe_module_file_modified, Filename, > > TopLevelPCHName); > > + Error(diag::err_fe_module_file_modified, Filename, TopLevelPCHName, > > + (unsigned)FileChange); > > else > > - Error(diag::err_fe_ast_file_modified, Filename, TopLevelPCHName); > > + Error(diag::err_fe_ast_file_modified, Filename, TopLevelPCHName, > > + (unsigned)FileChange); > > > > // Print the import stack. > > if (ImportStack.size() > 1 && !Diags.isDiagnosticInFlight()) { > > @@ -5192,6 +5253,8 @@ bool ASTReader::readASTFileControlBlock( > > consumeError(MaybeRecordType.takeError()); > > } > > switch ((InputFileRecordTypes)MaybeRecordType.get()) { > > + case INPUT_FILE_HASH: > > + break; > > case INPUT_FILE: > > bool Overridden = static_cast<bool>(Record[3]); > > std::string Filename = Blob; > > @@ -12153,7 +12216,7 @@ ASTReader::ASTReader(Preprocessor &PP, I > > StringRef isysroot, bool DisableValidation, > > bool AllowASTWithCompilerErrors, > > bool AllowConfigurationMismatch, bool > > ValidateSystemInputs, > > - bool UseGlobalIndex, > > + bool ValidateASTInputFilesContent, bool > > UseGlobalIndex, > > std::unique_ptr<llvm::Timer> ReadTimer) > > : Listener(DisableValidation > > ? cast<ASTReaderListener>(new > > SimpleASTReaderListener(PP)) > > @@ -12167,6 +12230,7 @@ ASTReader::ASTReader(Preprocessor &PP, I > > AllowASTWithCompilerErrors(AllowASTWithCompilerErrors), > > AllowConfigurationMismatch(AllowConfigurationMismatch), > > ValidateSystemInputs(ValidateSystemInputs), > > + ValidateASTInputFilesContent(ValidateASTInputFilesContent), > > UseGlobalIndex(UseGlobalIndex), > > CurrSwitchCaseStmts(&SwitchCaseStmts) { > > SourceMgr.setExternalSLocEntrySource(this); > > > > > > Modified: cfe/trunk/lib/Serialization/ASTWriter.cpp > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Serialization/ASTWriter.cpp?rev=374841&r1=374840&r2=374841&view=diff > > ============================================================================== > > --- cfe/trunk/lib/Serialization/ASTWriter.cpp (original) > > +++ cfe/trunk/lib/Serialization/ASTWriter.cpp Mon Oct 14 16:02:03 2019 > > @@ -1099,6 +1099,7 @@ void ASTWriter::WriteBlockInfoBlock() { > > > > BLOCK(INPUT_FILES_BLOCK); > > RECORD(INPUT_FILE); > > + RECORD(INPUT_FILE_HASH); > > > > // AST Top-Level Block. > > BLOCK(AST_BLOCK); > > @@ -1764,6 +1765,7 @@ struct InputFileEntry { > > bool IsTransient; > > bool BufferOverridden; > > bool IsTopLevelModuleMap; > > + uint32_t ContentHash[2]; > > }; > > > > } // namespace > > @@ -1787,6 +1789,13 @@ void ASTWriter::WriteInputFiles(SourceMa > > IFAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // File name > > unsigned IFAbbrevCode = Stream.EmitAbbrev(std::move(IFAbbrev)); > > > > + // Create input file hash abbreviation. > > + auto IFHAbbrev = std::make_shared<BitCodeAbbrev>(); > > + IFHAbbrev->Add(BitCodeAbbrevOp(INPUT_FILE_HASH)); > > + IFHAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); > > + IFHAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); > > + unsigned IFHAbbrevCode = Stream.EmitAbbrev(std::move(IFHAbbrev)); > > + > > // Get all ContentCache objects for files, sorted by whether the file is > > a > > // system one or not. System files go at the back, users files at the > > front. > > std::deque<InputFileEntry> SortedFiles; > > @@ -1810,6 +1819,25 @@ void ASTWriter::WriteInputFiles(SourceMa > > Entry.BufferOverridden = Cache->BufferOverridden; > > Entry.IsTopLevelModuleMap = isModuleMap(File.getFileCharacteristic()) > > && > > File.getIncludeLoc().isInvalid(); > > + > > + auto ContentHash = hash_code(-1); > > + if (PP->getHeaderSearchInfo() > > + .getHeaderSearchOpts() > > + .ValidateASTInputFilesContent) { > > + auto *MemBuff = Cache->getRawBuffer(); > > + if (MemBuff) > > + ContentHash = hash_value(MemBuff->getBuffer()); > > + else > > + // FIXME: The path should be taken from the FileEntryRef. > > + PP->Diag(SourceLocation(), diag::err_module_unable_to_hash_content) > > + << Entry.File->getName(); > > + } > > + auto CH = llvm::APInt(64, ContentHash); > > + Entry.ContentHash[0] = > > + static_cast<uint32_t>(CH.getLoBits(32).getZExtValue()); > > + Entry.ContentHash[1] = > > + static_cast<uint32_t>(CH.getHiBits(32).getZExtValue()); > > + > > if (Entry.IsSystemFile) > > SortedFiles.push_back(Entry); > > else > > @@ -1834,17 +1862,26 @@ void ASTWriter::WriteInputFiles(SourceMa > > > > // Emit size/modification time for this file. > > // And whether this file was overridden. > > - RecordData::value_type Record[] = { > > - INPUT_FILE, > > - InputFileOffsets.size(), > > - (uint64_t)Entry.File->getSize(), > > - (uint64_t)getTimestampForOutput(Entry.File), > > - Entry.BufferOverridden, > > - Entry.IsTransient, > > - Entry.IsTopLevelModuleMap}; > > + { > > + RecordData::value_type Record[] = { > > + INPUT_FILE, > > + InputFileOffsets.size(), > > + (uint64_t)Entry.File->getSize(), > > + (uint64_t)getTimestampForOutput(Entry.File), > > + Entry.BufferOverridden, > > + Entry.IsTransient, > > + Entry.IsTopLevelModuleMap}; > > + > > + // FIXME: The path should be taken from the FileEntryRef. > > + EmitRecordWithPath(IFAbbrevCode, Record, Entry.File->getName()); > > + } > > > > - // FIXME: The path should be taken from the FileEntryRef. > > - EmitRecordWithPath(IFAbbrevCode, Record, Entry.File->getName()); > > + // Emit content hash for this file. > > + { > > + RecordData::value_type Record[] = {INPUT_FILE_HASH, > > Entry.ContentHash[0], > > + Entry.ContentHash[1]}; > > + Stream.EmitRecordWithAbbrev(IFHAbbrevCode, Record); > > + } > > } > > > > Stream.ExitBlock(); > > > > Added: cfe/trunk/test/Modules/validate-file-content.m > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Modules/validate-file-content.m?rev=374841&view=auto > > ============================================================================== > > --- cfe/trunk/test/Modules/validate-file-content.m (added) > > +++ cfe/trunk/test/Modules/validate-file-content.m Mon Oct 14 16:02:03 2019 > > @@ -0,0 +1,33 @@ > > +// REQUIRES: shell > > +// > > +// Check driver works > > +// RUN: %clang -fmodules -fsyntax-only > > -fmodules-validate-input-files-content %s -### 2>&1 | FileCheck > > --check-prefix=CHECK-CC1 %s > > +// CHECK-CC1: -fvalidate-ast-input-files-content > > +// > > +// PCH+Modules: Test that a mtime mismatch without content change is fine > > +// RUN: rm -rf %t > > +// RUN: mkdir %t > > +// RUN: echo '// m.h' > %t/m.h > > +// RUN: echo '#include "m.h"' > %t/a.h > > +// RUN: echo 'module m { header "m.h" }' > %t/module.modulemap > > +// RUN: %clang_cc1 -emit-pch -fmodules-cache-path=%t/cache -fmodules > > -fimplicit-module-maps -o %t/a.pch -I %t -x objective-c-header %t/a.h > > -fvalidate-ast-input-files-content > > +// RUN: touch -m -a -t 202901010000 %t/m.h > > +// RUN: %clang_cc1 -fsyntax-only -fmodules-cache-path=%t/cache -fmodules > > -fimplicit-module-maps -I %t -include-pch %t/a.pch %s -verify > > -fvalidate-ast-input-files-content > > +// > > +// PCH+Modules: Test that a mtime mismatch with content change > > +// RUN: rm -rf %t > > +// RUN: mkdir %t > > +// RUN: echo '// m.h' > %t/m.h > > +// RUN: echo '#include "m.h"' > %t/a.h > > +// RUN: echo 'module m { header "m.h" }' > %t/module.modulemap > > +// RUN: %clang_cc1 -emit-pch -fmodules-cache-path=%t/cache -fmodules > > -fimplicit-module-maps -o %t/a.pch -I %t -x objective-c-header %t/a.h > > -fvalidate-ast-input-files-content > > +// RUN: echo '// m.x' > %t/m.h > > +// RUN: touch -m -a -t 202901010000 %t/m.h > > +// RUN: not %clang_cc1 -fsyntax-only -fmodules-cache-path=%t/cache > > -fmodules -fimplicit-module-maps -I %t -include-pch %t/a.pch %s > > -fvalidate-ast-input-files-content 2> %t/stderr > > +// RUN: FileCheck %s < %t/stderr > > +// > > +// CHECK: file '[[M_H:.*[/\\]m\.h]]' has been modified since the > > precompiled header '[[A_PCH:.*/a\.pch]]' was built: content changed > > +// CHECK: '[[M_H]]' required by '[[M_PCM:.*[/\\]m.*\.pcm]]' > > +// CHECK: '[[M_PCM]]' required by '[[A_PCH]]' > > +// CHECK: please rebuild precompiled header '[[A_PCH]]' > > +// expected-no-diagnostics > > > > Added: cfe/trunk/test/PCH/validate-file-content.m > > URL: > > http://llvm.org/viewvc/llvm-project/cfe/trunk/test/PCH/validate-file-content.m?rev=374841&view=auto > > ============================================================================== > > --- cfe/trunk/test/PCH/validate-file-content.m (added) > > +++ cfe/trunk/test/PCH/validate-file-content.m Mon Oct 14 16:02:03 2019 > > @@ -0,0 +1,29 @@ > > +// REQUIRES: shell > > +// > > +// Check driver works > > +// RUN: %clang -x objective-c-header -fsyntax-only > > -fpch-validate-input-files-content %t/a.h -### 2>&1 | FileCheck > > --check-prefix=CHECK-CC1 %s > > +// CHECK-CC1: -fvalidate-ast-input-files-content > > +// > > +// PCH only: Test that a mtime mismatch without content change is fine > > +// RUN: rm -rf %t > > +// RUN: mkdir %t > > +// RUN: echo '// m.h' > %t/m.h > > +// RUN: echo '#include "m.h"' > %t/a.h > > +// RUN: %clang_cc1 -emit-pch -o %t/a.pch -I %t -x objective-c-header > > %t/a.h -fvalidate-ast-input-files-content > > +// RUN: touch -m -a -t 202901010000 %t/m.h > > +// RUN: %clang_cc1 -fsyntax-only -I %t -include-pch %t/a.pch %s -verify > > -fvalidate-ast-input-files-content > > +// > > +// PCH only: Test that a mtime mismatch with content change > > +// RUN: rm -rf %t > > +// RUN: mkdir %t > > +// RUN: echo '// m.h' > %t/m.h > > +// RUN: echo '#include "m.h"' > %t/a.h > > +// RUN: %clang_cc1 -emit-pch -o %t/a.pch -I %t -x objective-c-header > > %t/a.h -fvalidate-ast-input-files-content > > +// RUN: echo '// m.x' > %t/m.h > > +// RUN: touch -m -a -t 202901010000 %t/m.h > > +// RUN: not %clang_cc1 -fsyntax-only -I %t -include-pch %t/a.pch %s > > -fvalidate-ast-input-files-content 2> %t/stderr > > +// RUN: FileCheck %s < %t/stderr > > +// > > +// CHECK: file '[[M_H:.*[/\\]m\.h]]' has been modified since the > > precompiled header '[[A_PCH:.*/a\.pch]]' was built: content changed > > +// CHECK: please rebuild precompiled header '[[A_PCH]]' > > +// expected-no-diagnostics > > > > > > _______________________________________________ > > cfe-commits mailing list > > cfe-commits@lists.llvm.org > > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
-- Bruno Cardoso Lopes http://www.brunocardoso.cc _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
